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(57) Abstract: A reversible watermarking method embeds (115) 
auxiliary data (114) into a data set (1 10), such as an image, audio, 
video or other data, in a manner that enables full recovery of the 
original, un-modified data set. This method may be used to de- 
termine whether the data set has been tampered with. To improve 
embedding capacity without the need for compression of the aux- 
iliary data, the method uses an expansion technique. One particu- 
lar approach exploits the correlation or redundancy within the data 
set to convert the data to a set of small, expandable values, such 
as difference values. These small values are then expanded by in- 
serting auxiliary data as one or more additional bits, increasing the 
number of bits without causing an underflow or overflow. This 
approach also uses a property of the data set that is invariant to 
the embedding operation (112A) to identify embedding locations 
(113), obviating the need for separate data to identify where data 
is embedded in a data set. 
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1 Reversible Watermarking 

2 

3 Related Applications; 

4 This application claims the benefit of provisional application 60/404,181 , filed 

5 August 16, 2002, 60/340,651, filed December 13, 2001, and 60/ , , filed 

6 December 2, 2002, entitled Reversible Watermarking by Jun Tian and Steve 

7 Decker. 
8 

9 This application is also related to application 10/035,830 filed October 18, 

1 0 2001 , which claims the benefit of provisional applications: 

1 1 a) 60/247,389, filed November 8, 2000; 

12 b) 60/260,907, filed January 10, 2001; and 

13 c) 60/284,594 filed April 17, 2001. 

14 The entire content of the above listed applications is hereby incorporated 

15 herein by reference. 
16 

17 Field of the Invention: 

18 The invention relates to steganography, auxiliary data embedding in data sets, 

19 and digital watermarks. 
20 

21 Background and Summarv 

22 The technology for digital watermarking media content, such as images, video 

23 and audio is well known. A variety of different types of digital watermarks have 

24 been developed. Some types of digital watermarks can be read from 

25 watermarked data despite changes in the data. For example, some types of 

26 image watermarks can survive when the watermarked image is rotated, 

27 spatially scaled, lossily compressed, and/or printed. Some video and audio 

28 watermarks survive when the watermarked content is lossily compressed, 

29 converted to analog form, and re-sampled Into digital form. 
30 

31 Some digital watermarks are designed to be fragile so that if the watermarked 

32 data is changed the watermark is rendered unreadable or is degraded in a 
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1 predictable fashion. Such watermarks can be used to determine if a 

2 watermarked document has been changed based on detection of the digital 

3 watermark. If certain data is watermarked with a fragile watermark, and the 

4 data is later changed the watermark is degraded or rendered unreadable. 

5 Thus, the absence or degradation of a watermark will indicate that the data has 

6 been changed. 
7 

8 Some digital watermarks are designed to be reversible. A watermark Is 

9 reversible if a data set can be watermarked, thereby changing the data 

10 somewhat, and at a later time the watermark can be removed in order to return 

11 to the original un-watermarked data set. 
12 

13 The technique used to watermark an image (or data set) determines such 

14 factors as: the extent to which a watermark can survive changes in an image, 

15 the amount of change in an image needed to destroy a fragile watermark, and 

16 how accurately an image can be recreated after a reversible watermark is 

17 removed. 
18 

19 One challenge that occurs with some reversible watermarks is that they can 

20 cause overflow or underflow conditions. For example, consider a digital image 

21 or audio signal that Is represented by values from 0 to 255. If during the digital 

22 watermarking operation, a digital sample with the value of 254 is increased by 

23 2, there will be an overflow condition. Likewise, if a sample with a value of 1 is 

24 decreased by 2, an underflow condition will occur. When an overflow or 

25 underflow occurs during a watermarking operation, it poses limitations on the 

26 ability to recover the original, un-watermarked signal. 
27 

28 The invention provides a number of methods and related software and systems 

29 for embedding auxiliary data in data sets, and for decoding this auxiliary data 

30 from the data sets. One aspect of the invention is a method of reversibly 

31 embedding auxiliary data in a data set. This method transforms the data set 

32 from an original domain into transformed data values with an invertible 
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1 transform. It expands selected data values to ennbed auxiliary data. The 

2 method then inverts the transformed data values, inciuding the data values 

3 selected for expansion, to return the transformed data values to the original 

4 domain. 
5 

6 Another aspect of the invention is a compatible decoder for extracting the 

7 embedded data and restoring the values of the data set to the same values as 

8 before embedding of the auxiliary data. This decoder transforms the data set 

9 from an original domain into transformed data values with an invertible 

1 0 transform. It extracts auxiliary data from data values previously selected for 

1 1 embedding of auxiliary data by expansion, and restores the selected data 

12 values to the same values as before the embedding of the auxiliary data. It 

1 3 then inverts the transformed data values, including the data values selected for 

14 expansion, to return the transformed data values to the original domain. 
15 

16 Another aspect of the invention is a method of reversibly embedding auxiliary 

1 7 data in a data set. This embedding method selects embedding locations in the 

1 8 data set that have a property that is invariant to changes due to embedding of 

19 the auxiliary data. The invariant property enables a decoder to identify 

20 embedding locations. The embedding method then reversibly embeds auxiliary 

21 data into data values at the embedding locations. 
22 

23 Another aspect of the invention is a method of decoding reversibly embedded 

24 auxiliary data in a data set. This method identifies a subset of locations in the 

25 data set that have a property that is invariant to changes due to embedding of 

26 the auxiliary data. It extracts auxiliary data from data values at the identified 

27 locations. It then restores values of the data set to the same values as before 

28 the embedding of the auxiliary data into the data set. 

29 Another aspect of the invention is a method of embedding auxiliary data in a 

30 data set. This method identifies values derived from the data set that are 

31 expandable. It expands the identified values by inserting an auxiliary data state 

32 corresponding to auxiliary data to be embedded in the identified values. This 
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1 method has a corresponding decoding method, and can be used for reversible 

2 data embedding applications. 
3 

4 Further features will become apparent from the following detailed description 

5 and accompanying drawings. 
6 

7 

8 Brief Description of the Drawings: 

9 Figure 1A is a diagram illustrating an expansion method for auxiliary data 

10 embedding into a data set. 

1 1 Figure 1 B is a diagram illustrating an auxiliary data decoder compatible with the 

12 data embedding method of Fig. 1A 

13 Figure 1C is a diagram illustrating an embedding operation for authentication 

14 applications. 

15 Figure 1 D is a diagram illustrating authentication by extracting the embedded 

16 data, re-creating the original data, and using the embedded data to 

1 7 authenticate the data. 

1 8 Figure 1 E is a diagram illustrating a reversible watermarking method used to 

19 select elements for embedding based on whether the element has a property 

20 that is invariant to the embedding operation. 

21 Figure IF is a diagram illustrating the decoding of a reversible watermark that 

22 takes advantage of the invariant property to identify embedded data locations. 

23 Figure 2A is a diagram of an image showing a pattern of bit pairs. 

24 Figure 2B is a diagram illustrating changeable and unchangeable bits in 

25 difference values. 

26 Figure 3 is an overall block flow diagram of the watermark embedding process. 

27 Figure 4 is a block flow diagram of the watermark reading process. 
28 

29 Detailed Description: 

30 Various preferred embodiments of the invention will be described. The 

31 embodiments provide a method or technique for embedding a digital watermark 

32 into a data set, such as an image. Embodiments illustrate a reversible 



4 



wo 03/055130 



PCTAJS02/40162 



1 watermarking method that enables decoding of the digital watermark, and exact 

2 re-creation of the original, un-watermarked data. 
3 

4 While certain embodiments described below relate to digital watermarking of 

5 image signals, the invention can be used to watermark other types of data such 

6 as audio data. 
7 

8 Figure 1 A illustrates a flow diagram of an expansion method for auxiliary data 

9 embedding into a data set. This particular method is designed to be invertible 

10 in cases where there are no changes to the data set (e.g., "fragile" data 

1 1 embedding). Variations of the method may be designed to make the data 

12 method more robust to certain types of changes to the data set and partially 

13 reversible. For example, the method may be employed hierarchically to 

14 transformations of the data set into layers of values that have varying 

15 robustness. 
16 

17 As illustrated in Fig. 1A, the embedder starts with a data set 20. For 

1 8 applications that we are targeting, this data set comprises a set of integers 

19 (e g-, 8 bit values ranging from 0-255). The embedder performs an integer to 

20 integer transform of the data into values for expansion (22). This transform 

21 maps sets of data elements in the data set into values for expansion. The 

22 embedder applies this transform across the entire data set to be embedded 

23 with auxiliary data (e.g., it is repeated on groups of elements throughout the 

24 data set). Note that in some applications, the data may undergo one or more 

25 pre-processing steps to place the data into a better format for the data 

26 embedding method. 
27 

28 The specific type of transform may vary, and the implementer may select the 

29 transform for the needs of the application. One of our applications of the 

30 method is reversible digital watermark embedding for images. Our criteria 

31 include making the embedding operation perfectly reversible, maintaining (or at 

32 least controlling to a desired degree) the perceptual quality of the image signal, 
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1 and embedding capacity of the digital watermark. In other applications, other 

2 objectives may be important, such as retaining some level of lossless 

3 compressibility of the embedded data, enhancing the security of the embedding 

4 process (e.g., making the nature of the transform statistically undetectable), 

5 etc. 
6 

7 In our specific embodiments, the embedder transforms sets of integer data to 

8 corresponding sets of values for expansion, including fixed and variable values. 

9 The fixed values remain unchanged in the subsequent expansion embedding 

10 operation. The variable values are selected for expansion to serve as carriers 

11 of the embedded data. We selected a transform that generates fixed values 

12 that enables reversibility and perceptual quality control. We also selected this 

1 3 transform because it generates small integer variable values that are likely to 

14 be more expandable to provide for higher information carrying capacity. The 

1 5 specific transform is a transform of sets of the data into corresponding sets of 

16 averages and difference values. Other transforms that satisfy the criteria may 

17 be selected as well. 
18 

1 9 Next, the embedder performs an invertible expansion of values in the sets of 

20 values transformed for expansion (24). This expansion is referred to as 

21 invertible because it enables the auxiliary data decoder to extract the 

22 embedded data values for each set, and compute the original data values 

23 computed for expansion in the embedder. 
24 

25 The sets of data include two or more data elements. The embedder transforms 

26 these data elements into a corresponding set of values for expansion. The 

27 embedder embeds auxiliary data by expanding selected values for expansion 

28 in this set into expanded values that represent auxiliary data. The auxiliary 

29 data may be binary or higher state (e.g., two or more possible states for the 

30 embedded data value). 
31 



6 



wo 03/055130 



PCT/US02/40162 



1 In the case of the transform to sets of fixed and variable values, the embedder 

2 expands the variable values into expanded values that carry the binary or 

3 higher embedded state. The expansion operation multiplies a value for 

4 expansion by an integer corresponding to the number of states and adds the 

5 desired state. 
6 

7 Here are examples of expanding an integer, I, using a two or more state 

8 expansion operation: 

9 Two states: 

10 21+0 

11 21+1 
12 

13 Three States: 

14 31+0 

15 31+1 

16 31+2 
17 

18 N States: 

19 NI+0 

20 NI+1 

21 NI+2 

22 

23 . 

24 NI+(N-1) 

25 

26 Next, the embedder performs the inverse of the transform in block 22 on the 

27 sets of values, including expanded values (26). This inverse transform returns 

28 the embedded data set 28 back to its original domain at the input of the 

29 process. 
30 

31 Figure IB illustrates the corresponding auxiliary data decoder. First, the 

32 decoder performs the same transform as in block 22 to place the data Into the 
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domain where it was expanded (30). Next, tiie decoder extracts the auxiliary 
data values by performing the inverse of the Invertible expansion operation 
(32). In the case where the expansion multiplies by the number of states and 
adds the desired state, the decoder extracts the embedded data value directly 
by reading the state that has been added to the expanded value. This inverse 
of the expansion provides the original un-expanded value as well as the 
embedded data value. 

Having recovered the un-expanded value in the set, the decoder now performs 
the inverse transform (34) as in block 26 to get the original data set 36. 

To help illustrate, we show examples of this method in mathematical form. 
First, we illustrate an example of a transform of data elements, pi, p2, and p3, 
into values for expansion a, dj, and d2. 

Generally, the transformation involves two or more elements of the data set into 
the values for potential expansion. In this case, we illustrate a transform 
involving three elements of the data set: 



a 




r 


~ Px 










Pi 








V 




J 



A specific example of the function /is: 

P} -^Pl +P3 



a = 



where [ J, is the least Integer function. 

For embedding data in digital images, the data elements correspond to discrete 
image samples, such as pixels in the spatial domain of the image. In this 
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1 example, one can see that the value, a, comprises an average of the elements, 

2 while di and d2 comprise difference values of selected pairs of the elements. 

3 The average may be weighted differently. For images, the data samples may 

4 correspond to grayscale values, or for color images, the samples may 

5 correspond to luminance, chrominance, or a selected combination of samples 

6 from some other color channel or color mapping. As an example, the color 

7 components R, G, B or CMY, may be uncorrected before embedding and then 

8 independently embedded. Alternatively, the transform A may compute the 

9 fixed value as a function of the RGB values: (R+2G+B)/4, for example. 
10 

1 1 Though not a requirement, this transformation shows an example of a case 

12 where the transform produces fixed and variable values: a remains fixed in the 

1 3 expansion operation, while dj and d2 are potentially expanded. 
14 

15 This example illustrates that the data elements in the set, and their 

16 arrangement in the original data set may vary. In the case where the 

17 implementer is seeking better embedding capacity, the data elements are 

18 preferably selected to provide highly expandable values. In an invertible 

19 expansion method, smaller values are preferable because they can be 

20 expanded further before causing a non-invertible exception, namely, an 

21 underflow or overflow of the data elements, which are constrained to a 

22 predetermined range of integers. 
23 

24 In the case of digital data, such as 8 bit values, the values are constrained to a 

25 range of integers such as 0 to 255. In the case of digital image pixels that are 

26 transformed into fixed average values and expandable differences, highly 

27 correlated pixel values provide the smallest difference values, and as such, are 

28 more expandable. Thus, selecting a pattern of neighboring data elements 

29 tends to provide groups of correlated elements, whose difference values are 

30 more expandable. 
31 
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The 2"^ and 3'''' equations representing the transformation are merely functions 
that give small numbers that are expandable. The difference between two 
correlated values is just one example. Anther example is the difference 
between a data element and some fixed value such as 0 or 255. By varying 
the transform function adaptively throughout the data set, the embedder can 
optimize the capacity, perceptibility, or some other combination of criteria. To 
inform the decoder of the proper function selected at embedding, the embedder 
may base the selection of the function based on data element features that are 
invariant to the embedding operation, or it may make the Identification of the 
function part of the key used to decode the embedded data. 

Next, to Illustrate data embedding through expansion in this example, consider 
the following expression: 



, where is the inverse function of / as shown in the 



following example 

p, =a 

Pj-dj- 
P2 =^2 -Pi 

E is the expansion matrix as shown in the following example: 
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in this example, the first row of the expansion matrix illustrates that a is the 
fixed value, while the next two rows represent functions that expand values dj 
and d2 as a function of the number of states, N, and the desired state of the 
symbol to be embedded, s. The number of states per expandable value is 
variable. The total number, M(3 in the example above), of data elements, je?, is 
also variable in function / 
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1 The total embedding capacity per grouping of elennents, p, in the function / can 

2 be represented as: 

3 {MA))Log2N bits; and the capacity per data element, p can be 

4 represented as: 

5 {(MA)IM)Log2Nb\\s 
6 

7 As shown in this example, the transformation of the expanded data by the 

8 inverse of/ produces the embedded data set, pj \ p2 ' and ps \ 
9 

10 For reversibility, the embedder preferably uses invertible integer to integer 

1 1 transforms. In our implementation, we use the floor function to ensure that the 

12 functions, /and E, are integer to integer and invertible. 
13 

14 The methods outlined above may be repeated on the data set to embed 

15 additional layers of auxiliary data, each possibly with a different decoding key 

16 used to enable decoding of the layer. Specifically, the input of one embedding 

17 operation may produce an embedded data set that is input to another 

18 embedding operation. This embedding may be performed repeatedly and 

19 hierarchically to embed additional data. A hierarchical approach applied to 

20 expandable values in different transform domains of varying robustness can 

21 provide an embedding scheme that is robust and reversible in part. One 

22 example would be to apply the method hierarchically to different spatial 

23 resolutions of an image. For example, the implementer may seek to embed 

24 data by expanding the difference of average values, which are more robust to 

25 distortion. 
26 

27 As the implementer seeks to improve the performance of the data embedding 

28 to optimize capacity, perceptual quality, robustness, detectability, etc., the 

29 domain of the data set and the transform of the data set to values for expansion 

30 may be selected to optimize the desired performance criteria. 
31 
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1 As the implementer seeks to make the data embedding more robust, there are 

2 tradeoffs with embedding capacity and being able to achieve perfect 

3 reversibility. If the embedded data must survive certain types of distortion, the 

4 distortion may preclude reversibility of all or a portion of the data that is 

5 embedded in attributes that are altered by the distortion. Conversely, unaltered 

6 robust attributes that carry the embedded data can remain reversible. 
7 

8 In general, to Increase robustness, the implementer can select a pre- 

9 processing operation on the data set that transforms it into a domain that is 

10 more robust to the expected forms of distortion. For example, if some loss of 

1 1 the original data were tolerated, the original data set may be pre-quantized with 

12 more coarse quantization before applying the data embedding method. Also, 

13 while our examples focus on spatial domain pixels, the data embedding method 

14 applies to other domains such as wavelet, DCT, Fourier, etc. 
15 

16 One observation of the example transform of data to fixed averages and 

17 expandable differences is that a lower resolution thumbnail image may be 

18 computed using the average function. In this case, the thumbnail of the 

19 watermarked and un-watermarked image computed by this average function 

20 are the same. 
21 

22 For images, the method may be repeated on contiguous tiles of pixels, each 

23 embedded with its own reference code that enables the data to be robust to 

24 cropping. 
25 

26 Figures 1C and 1D shows compatible embedder and decoder processes that 

27 ensure there is no difference between the original data set and the re-created 

28 data set. The process begins with an original data set 101. As indicated by 

29 block 102, the embedder calculates authentication data, such as a hash of the 

30 original data, error detection data, a fixed message, or an error correction 

31 encoded message that can be analyzed to detect the presence of errors in the 

32 embedded data. As indicated by block 103, the embedder embeds auxiliary 
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1 data in the data set 101, including the authentication data along with other 

2 auxiliary data. The embedded data set is designated 104. 
3 

4 When one wants to recreate the original data set, the embedded data set 1 04 

5 is processed as indicated by block 105 to read the embedded auxiliary data. 

6 Processes used to read the auxiliary data are explained further below. The 

7 authentication data and various other auxiliary data are extracted from the 

8 embedded data set 104. The extracted data is used to re-create the original 

9 data set from the embedded data set as indicated by block 106. Finally, the 

10 reader uses the authentication data to check whether the re-created data set is 

1 1 unmodified (e.g., the same as the original data set). For example, a new hash 

12 number X2 is calculated from the re-created data set. If the hash number X2 

13 equals the embedded hash X, it means that the original data set and the re- 

14 created data set are identical. 
15 

1 6 Alternatively, an error detection message can be used to detect whether the 

17 extracted auxiliary data is error free, which is expected if the embedded data 

18 set has not been modified. Other fixed data messages in the auxiliary data can 

19 be checked for errors by comparison with a known, expected message. 

20 Finally, an error corrected version of embedded data may be used to 

21 regenerate a new error correction encoded message, which is then compared 

22 with the extracted, error correction encoded message to check for errors. 
23 

24 In some applications, it is useful to be able to identify where auxiliary data is 

25 embedded in an embedded data set using only the embedded data (e.g., 

26 without a map separate from the embedded data). One approach to 

27 accomplish this is to identify and embed at least some of the auxiliary data in 

28 embedding locations that are identifiable before and after the embedding 

29 operation. In particular, certain features can be selected that are invariant to 

30 the embedding operation and serve to identify an embedding location. These 

31 features enable the auxiliary data decoder to identify variable embedding 

32 locations by finding the location of features with the invariant property. 
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1 

2 Figure 1 E illustrates an embedding method that identifies data elements in an 

3 image that are invariant to auxiliary data embedding to enable the decoder to 

4 locate the embedded data. A similar approach may be used for embedding 

5 auxiliary data in other data types. First, as indicated by block 1 1 1 , an optional 

6 transfomri is applied to an original image 1 10 to produce a transformed image 

7 112. One example of this transform 111 calculates difference and average 

8 values for pairs of pixels in an image. Next as indicated by blocks 113, certain 

9 elements in the transformed image 112 are identified. The identified elements 

10 have a property that remains identifiable after they are changed by auxiliary 

1 1 data embedding. The identified elements are illustrated as blocks 112A. It 

12 should be understood that in a practical application, an image has many 

1 3 thousand of such elements. For convenience of illustration, only a few such 

14 elements 1 12A are illustrated in Figure IE. 
15 

16 An auxiliary data stream 1 14 is embedded in the image. The auxiliary data 

17 stream can include authentication data, payload data, and various other data 

1 8 elements. As indicated by block 115, the data stream 1 14 is embedded in the 

1 9 elements 1 12A of image 112 creating a new image 116, which has identifiable 

20 elements 1 16A. The elements 1 12A and the elements 1 1SA have different 

21 values; however, they can be identified or picked out of all of the other 

22 elements in images 112 and 116, because the selection criteria uses a property 

23 which is invariant between the original elements and the elements that have 

24 been changed by the embedding process. 
25 

26 The embedding locations having the invariant property may be used to embed 

27 auxiliary data, such as a location map. that identifies further embedding 

28 locations. 
29 

30 Some embodiments of reversible watermarking embed values of the original 

31 image that are changed by bit substitution during the embedding operation as 

32 part of the auxiliary data stream. This is not required in all cases because 
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1 some embedding operations, like the expansion embedding method, are 

2 invertible without storing original data values and can be made at some 

3 locations in a manner that retains the invariant property. 
4 

5 An inverse transform 1 17 (I. e. a transform that is the inverse of the transform 

6 111) can be applied to image 1 16 to generate an embedded image 118 (i.e. an 

7 image with the auxiliary data embedded in it). The image 1 18 is shown with a 

8 shaded corner to indicate that image 118 includes embedded auxiliary data. 
9 

10 The auxiliary data reading and image re-creation process is illustrated in block 

1 1 diagram form in Figure 1F. First as illustrated by block 121, a transform is 

12 applied to the image with embedded data 118. The transform 121 is identical 

13 to the transform 111. Application of transform 121 produces a transformed 

14 image 116, which has identifiable elements 1 16A. These elements are 

15 identified using the same invariant criteria 123. As indicated by block 125, the 

16 data stream 1 14 is extracted from elements 1 16A. As indicated by block 126, 

17 the data from stream 1 14 is used to restore the transformed values of the 

1 8 image to their original values prior to auxiliary data embedding. In certain 

19 cases, this process of restoring the original values of the transformed data 

20 occurs as part of the auxiliary data extraction process of block 125. In other 

21 cases, certain changed bit values of elements 1 16A are replaced with original 

22 bit values carried in the auxiliary data stream. It is not necessary to carry 

23 original values of the image data in the auxiliary data stream when using 

24 embedding techniques, like expansion, that are invertible without requiring the 

25 auxiliary data to include the changed bits of the original image. As indicated by 

26 block 127, an inverse of transform 121 is applied to re-generate the original 

27 image now which is designated 1 10A in Figure IF. Not specifically shown in 

28 Figure 1 F, is the fact that data stream 1 14 can include a hash of the original 

29 image 110. One can generate a hash of the recreated original image 1 1 0A and 

30 compare it to the hash in data stream 1 14, to insure or guarantee that the 

31 image has been re-created precisely. 
32 
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1 In certain embodiments of our reversible watermarking method, an invertible 

2 transform divides the pixels in an image into pairs or groups of pairs according 

3 to a particular pattern. Factors to be considered in choosing these patterns 

4 include, for example, retaining perceptual quality of the image after embedding, 

5 increasing data capacity, etc. Figure 2A illustrates (in greatly exaggerated 

6 form) the individual pixels in an image. Only a small portion of an image is 

7 shown. As is well known, any practical image would include many thousand 

8 such pixels. For convenience of illustration only a relatively few pixels are 

9 shown in Figure 2A. It is also noted that in certain embodiments only the 

10 luminance values of the pixels are embedded with data. That is, the image is 

1 1 viewed as a gray scale image. Naturally, in color images there would also be 

12 color values. It should be understood that the digital watermark could 

13 alternatively be placed in other aspects of the image such as in the various 

14 color components and other transform domain sample value, like frequency 

15 domain values. 
16 

17 The purpose of Figure 2A is to illustrate that the pixels are grouped into pairs in 

18 this example embodiment. For example, as shown in Figure 2A, pixel C and D 

19 belong to the same pair. Any pattern of grouping can be used; however, the 

20 same pattern must be used in both the embedding and in the reading 

21 operations. While any pattern of paired pixels can be used, it is advantageous 

22 to use pairs that probably have similar values, that is, pairs that probably will 

23 have small difference numbers. Thus, in the preferred embodiment, adjacent 

24 pixels were chosen for members of each pair. In Figure 2A, an alternating 

25 horizontal and vertical pattern was chosen to illustrate that the pattern can have 

26 a wide variety of arrangements. 
27 

28 In certain embodiments using difference expansion, two numbers are 

29 calculated for each pair of values in the image: 

30 a) The average value of the two pixels, and 

31 b) The difference between the values of the two pixels. 
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1 Transforming the image representation from a representation with an array of 

2 pixel values to a representation with an array of difference and average 

3 numbers is just one example of a transform or filter as indicated by block 1 1 1 in 

4 Figure 1E. Other transformations may be made before this transform to place 

5 the original data in a format for embedding in other domains (e.g., a transform 

6 to a frequency domain, a transform a feature set, such as autocorrelation 

7 values or other statistical values). 
8 

9 In order to facilitate a discussion of additional embodiments, the following terms 

10 are defined as follows: 



1 1 Average value: the average value of a group of two or more values. 

12 Difference value: the difference between selected values in the group 

13 Expandable value: a value that can be expanded without causing an 

1 4 overflow or underflow. 

15 Expanded value: a value that has been expanded. 

16 Changeable value: all expandable values and values that can be 

17 changed by bit substitution without causing an overflow or 

18 underflow. 
19 

20 These definitions are used only for the sake of explaining certain embodiments, 

21 and are not intended to be limiting. 

22 

23 Figure 2B illustrates difference values A to Z to show examples of the various 

24 types of difference values that can exist in an image. Difference values A and 

25 C are difference values that are not changeable. Difference values B and Z are 

26 changeable, but not expandable. They have certain bits designated Be and Zc 

27 that may be changed by bit substitution. Difference values D and E are 

28 expandable. 



29 

30 As a simple example consider the following. If a pair of pixels has grayscale 

31 values (61 ,76), the average value of the pair is 68.5 and the difference is 1 5. 
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Only the integer part of the average, namely 68, need be considered. This 
integer part is computed using the floor function, for example. The difference 
value 15 can be expressed as a binary number with a minimum length. In such 
a representation, all leading "0"s in the binary representation are discarded. 
That is, the difference number 15 can be expressed as the binary number 
1111. 

With this example, a bit can be inserted in the difference number 1111 without 
causing an overflow. That is, where the difference number is 1 1 1 1 and a 0 is 
inserted after the first 1 , the number becomes 101 1 1 or 23. 

Given an average of 68.5 and a difference of 23, the pair of pixels must have 
the value 57 and 80. The average of 57 and 80 is 68.5 and the difference is 
23. The above numbers may be easier to follow with the following table. 



Pixel 
values 


Average 


Difference 


Difference value 
in binary 


61, 76 


68.5 


15 


1111 


57, 80 


68.5 


23 


10111 



It is noted other pairs of pixels values could have an average of 68; however, 
only the values 57 and 80 have an average of 68 (ignoring the fractional 
portion) and a difference of 23. 

The following is another simple example to illustrate difference expansion. 
Assume that one has two grayscale values x = 205 and y = 200. We will 
illustrate below how one can embed one bit b = 1 , in a reversible way. First the 
integer average value / and the difference value "h" of x and y are computed 
as follows: 



/= 



x + y 




205 + 200 




405 


2 




2 




_ 2 . 



= 202 
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h = x-y = 205-200 = 5 
It is noted that the symbol \_ j\s the floor function meaning "the greatest 

integer less than or equal to". For example [2.7] = 2, and [-5.2] = -6 . 
Next we represent the difference number h in Its binary representation: 

h = 5 = IOI2 

Then we append b which equals 1 into the binary representation of h after the 
least significant bit (LSB), the new difference number h' will be: 

h' = 101b2 = IOII2 = 11 
The above Is equivalent to: 

h' = 2xh + b = 2x5+1= 11 
Finally we can compute the new grayscale values, based on the new difference 
number h' and the original average number / , 

= 208 



/+ 


h'+l 


= 202 + 


11 + 1 




2 




2 



y' = l- 



h' 


= 202- 


11 




_2. 




.2_ 



= 197 



From the embedded pair x\y', we can extract the embedded bit b and restore 
the original pair x, y. To do this we again compute the integer average and 
difference as follows: 



x'+y 




208 + 197 


2 




2 



= 202 



h' =x' -y' =208-197 = 11 

We now look at the binary representation of h' 
h' = 11 = 10II2 

From the above we extract the LSB, which in this case is 1 , as the embedded 
bit b which leaves the original value of the difference number as: 

h = IOI2 =5 
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1 the above is equivalent to: 

2 b = LSB(h') = 1, h= — = 5. 



2 



3 

4 With the original average value / and the restored difference number h, we can 

5 restore exactly the original grayscale valued pair, x,y. 
6 

7 In the above example, although the embedded pair (208, 197) is still 8 bits per 

8 pixel (bpp), one bit b has been embedded by increasing the valid bit length of 

9 the difference number h from 3 bits (for h = 5) to 4 bits (for h' = 1 1). This 

10 reversible data embedding operation h' = 2 x h + b is called difference 

1 1 expansion. 
12 

13 The reason that the valid bit length of the difference numbers h can be 

14 increased in images is because of the redundancy that exists in the pixel 

15 values of natural images. In most cases h will be very small and have a short 

16 valid bit length in its binary representation. However, in an edge area or an 

17 area containing lots of activity, the difference number h from a pair of grayscale 

18 values could be large. For example, if x = 105, and y = 22, then h = x-y = 83. 

19 In such a situation if one wanted to embed a bit 0 into h by difference 

20 expansion, then h' = 2 x h +b = 166. With / = 63 being unchanged, the 

21 embedded pair will be x' = 146 y' = -20. This will cause an underflow problem 

22 since grayscale values can only be in the range of [0, 255]. In the specific 

23 embodiments discussed below, the grayscale values selected for expansion 

24 are those grayscale values that can be expanded without causing an overflow 

25 or underflow condition. 
26 

27 The overall process used to watermark an image is illustrated in block diagram 

28 form in Figures 3 and the overall process used to read a watermark and re- 

29 create an image is illustrated in Figure 4. Each block in Figures 3 and 4 can be 

30 a subroutine in a program or digital circuit, or alternatively, a number of blocks 

31 can be performed by a single program subroutine or digital circuit. 
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1 

2 As indicated by blocic 300, the process begins with an image which one wants 

3 to embed auxiliary data (e.g., a digital watermark). It is noted that in other 

4 embodiments, one could start with other types of data. For example, instead of 

5 starting with an image, one might start with a digitized file of audio data, video 

6 data, software, graphical model (e.g., polygonal mesh), etc. 
7 

8 As a first step (block 301) a hash number or other authentication data is 

9 generated for the image. This can be calculated by known techniques for 

10 calculating a hash number. It is noted that the size of a hash is much smaller 

1 1 than the size of the image. It is not necessarily a unique identification. 

12 However, a hash can authenticate an image with a very high confidence level. 
13 

14 Block 302 indicates that a pattern of pixel pairs is selected. It is desirable (but 

15 not absolutely necessary) that the values in each pair tend to be similar. The 

16 selection pattern illustrated in Figure 2A is one example of selected pairs. 

17 Adjacent pairs have been selected since they more likely have relatively similar 

18 values. However, the particular pattern selected is arbitrary and a wide variety 

19 of different patterns could be used. 
20 

21 Next, as indicated by block 303, for each pair of pixels, two values are 

22 calculated. The average of the two pixel values of the pair is calculated and the 

23 difference between the pixel values in the pair is calculated. 
24 

25 The values of the pixel in each pair are then examined and the following is 

26 determined: 

27 a) Those pairs that can be expanded without causing an overflow or underflow. 

28 b) Those pairs that cannot be expanded, but which have bits that can be 

29 changed by bit substitution without causing an overflow or underflow, 

30 c) Those pairs that do not fall into groups "a" or "b." 
31 
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1 Various embodiments are described in detail below for selecting the 

2 expandable pairs. Note that the difference values of the pairs in sets "a" and 

3 "b" are both changeable in some fashion (by expansion or by bit substitution). 

4 The set of "changeable" difference values can be limited to those that have an 

5 invariant property to the embedding operation so that the decoder can identify 

6 embedding locations without use of data separate from the watermarked data. 
7 

8 As indicated by blocks 305 and 306, the particular pairs that will be expanded is 

9 determined and a location map is made which indicates which pairs will be 

10 expanded. For example, one simple way of making a location map is to have 

1 1 one bit for each pair that indicates whether the pair is expandable. Another 

12 way to make a location map is to store the index values of either the pairs that 

13 can be expanded or the indexes of the pairs that can not be expanded. 
14 

15 Next as indicated by block 307, a data stream (called the embedded data 

16 stream) is created. The embedded data stream may include: 

17 a) The desired payload data (i.e. data which one desires to store in the 

18 watermark). 

19 b) The location map (in some embodiments, the location map is compressed). 

20 c) The original bits changed by bit substitution, and 

21 d) A hash number of the original image. 
22 

23 As indicated by block 308, the embedder embeds the auxiliary data stream 

24 using expansion (and in some cases, bit substitution). For certain expandable 

25 difference values, the embedder expands the difference value by multiplying 

26 the difference value by the desired number of states and adding the desired 

27 state. For example, in the case of two states, the embedder multiplies the 

28 expandable difference value by 2, shifting the bit positions toward the MSB, 

29 and the embedded bit value (0 or 1) is added in the bit position vacated by the 

30 shift. As indicated by block 309, the new difference values along with the 

31 original average values are used to calculate new values for each pair. In 

32 certain cases, the embedder replaces bits in certain difference values (e.g.. 
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1 those in set "b") by certain bits from the embedded data stream using bit 

2 substitution. The result is a watermarked image 310. 
3 

4 Figure 4 shows auxiliary data decoder operations in the process of reading the 

5 auxiliary data and recreating the original, un-watermarked image. First as 

6 indicated by block 401 , the values in the watermarked image are grouped into 

7 pairs using the same pattern as was used during the watermarking process. 

8 Next (block 402) the average and difference value of the pairs are calculated. 
9 

10 The changeable difference values are determined (block 403). The decoder 

1 1 can identify these values using a property invariant to the embedding operation, 

12 - or using separate data (e.g., a separate location map). 
13 

14 As indicated by block 404, the changeable difference values are selected, and 

15 an auxiliary data stream is extracted. In this case, the auxiliary data embedded 

16 by expansion and by bit substitution Is carried In the LSBs of the difference 

17 values, and as such, is easily separated from the changeable difference values. 

18 This extracted data is the embedded data stream previously discussed. The 

19 embedded data stream includes: 



20 1)Thepayload 

21 2) The location map that tells which pairs have been expanded (if not 

22 provided separately). 

23 3) The original value of any bits, if any, changed by bit substitution 

24 4) a hash of the original image (or other authentication data). 



25 The length and position of each component in the embedded data stream is 

26 known (or it can be determined), hence, the embedded data stream can be 

27 separated into its component parts. 
28 

29 Block 406 indicates that the bits changed by bit substitution are replaced with 

30 the original bits in the embedded data stream. The location map is used to tell 

31 which pairs have been expanded. As indicated by block 406, the difference 

32 numbers for the pairs are processed in sequence. For each pair, any bits 
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1 changed by bit substitution are replaced by corresponding original bits from the 

2 embedded data stream. If the location map indicates that a particular pair was 

3 expanded, the difference values are restored to their original values by 

4 inverting the expansion operation. For the case of binary embedding states, 

5 this operation shifts the bit positions back to their original position. 
6 

7 Finally, new values for each pair are calculated from the average values and 

8 the restored difference values for each pair (block 407). These new values are 

9 the newly re-created image as indicated by block 408. 
10 

11 As a final step, a hash number for the re-created image is calculated and 

12 compared to the hash number that was in the embedded data stream. If the 

13 two numbers match, the original image has been re-created perfectly. 
14 

15 Several specific embodiments of the invention will now be described in 

16 considerable mathematical detail. It is noted that in the following discussion, 

17 some equations are referred to by the number in parentheses that is to the right 

18 of the equation. 
19 

20 Details of First Specific Preferred Embodiment: The following is a more 

21 detailed description of a first specific preferred embodiment of the invention. 

22 This embodiment provides a high capacity and high quality reversible 

23 watermarking method based on difference expansion. A feature of the method 

24 is that it does not involve compressing original values of the embedding area. 
25 

26 The method described here can be applied to digital audio and video as well. 

27 This embodiment performs steps similar to those in Figure 3. That is, the 

28 difference between neighboring pixel values are calculated (block 303). Some 

29 difference numbers are selected for difference expansion (block 305). The 

30 original values of difference numbers, the location of expanded difference 

31 numbers, and a payload are all embedded into the difference numbers (308). 

32 Extra storage space is obtained by difference expansion. 
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2 The described embodiment pertains to grayscale images. There are several 

3 options by which the technique can be applied to color images. One can de- 

4 correlate the dependence among different color components, and then 

5 reversibly watermark the de-correlated components. Or one can reversibly 

6 watermark each color component individually. 

7 

8 In this embodiment, a watermark is embedded in a digital image /, to create a 

9 watermarked image /'. The reversible watermark can be removed from /' to re- 

10 create the original image. The recreated image is called One can determine 

11 if the image /'was tampered with by some intentional or unintentional attack. 

12 This is done by comparing a hash of the original image / to a hash of the re- 

13 created image /" If there was no tampering, the retrieved image /" is exactly 

14 the same as the original image /, pixel by pixel, bit by bit. 

15 

16 The basic approach is to select an area of an image for embedding, and embed 

17 the payload. Difference expansion is used to embed the values in the image, 

18 and this eliminates the need for loss-less compression. The difference 

19 expansion technique discovers extra storage space by exploring the high 

20 redundancy in the image content. 

21 

22 This embodiment embeds the payload in the difference of neighboring pixel 

23 values. For a pair of pixels (x, y) in a grayscale image, 

24 x,y^Z,0<x,y< 255, we define their average and difference as 



25 / = 



.h = x-y (1) 



26 where the symbol [ J is the floor function meaning "the greatest integer less 

27 than or equal to". The inverse transform of (equation 1 above) is: 
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x = l + 



h + 1 


,y = l- 


h 






i 2 J 




.2. 



(2) 



As grayscale values are bounded in [0,255], we have: 

h + 1 



0</ + 



< 255,0 </ 



<255 



which is equivalent to: 

|/2|<min(2(255-/X2/ + l) 



(3) 



Thus to prevent overflow and underflow problems, the difference number h 
(after embedding) satisfies Condition (3). 

The least significant bit (LSB) of the difference number h will be the selected 
embedding area. As 



l + LSBQi) 



with LSB{h) = 0 or 1 , to prevent any overflow and underflow problems, we 
embed only in changeable difference numbers. 

Definition of Changeable values : For a grayscale-valued pair (x, y), we say h is 
changeable if: 



2 + b 



<min(2(255-/X2/ + l) 



for both b-0 and 1. 

Using bit substitution for changeable h does not provide additional storage 
space. We gain extra storage space from expandable difference numbers. 

Definition of Expandable values : For a grayscale-valued pair {x, y), we say h is 
expandable if 

\2'h + b\< min(2(255 - l\2l + 1) 
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1 for both b-0 and 1. 

2 In the binary representation of integers, an expandable h could add one extra 

3 bit b after its LSB, with 6=0 or 1 . More precisely, h could be replaced by a new 

4 difference number ^ -2/?+b, without causing an overflow or underflow. Thus, 

5 for each expandable difference number, one could gain one extra bit. The 

6 reversible operation from h to h' is called difference expansion. An expandable 

7 h \s also changeable. After difference expansion, the expanded h' is still 

8 changeable. 

g 

10 With this embodiment, more difference numbers will be changeable and/or 

1 1 expandable than in the fourth embodiment. Also note that if /7=0 or -1 , the 

12 conditions on changeable and expandable are exactly the same. 

13 

14 When this embodiment is applied to a digital image, the image is partitioned 

15 into pairs of pixel values. A pair comprises two pixel values or two pixels with a 

16 relatively small difference number. The pairing can be done horizontally, 

17 vertically, or by a key-based specific pattern. The pairing can be through all 

18 pixels of the image or just a portion of it. The integer transform (1) is applied to 

19 each pair, (it is noted that one can embed a payload with one pairing, then on 

20 the embedded image, we can embed another payload with another pairing, and 

21 so on.) 

22 After applying transform 1, five disjoint sets of difference numbers, EZ, NZ, EN, 

23 CNE, and NC are created: 

24 1 . EZ: expandable zeros. For all expandable h e {0,-l} 

25 2. NZ: not expandable zeros. For all not expandable h g {o -l} 

26 3. EN: expandable nonzeros. For all expandable h ^ {O-l} 

27 4. CNE: changeable, but not expandable. For all changeable, but not 

28 expandable h 0 {O -l} 
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1 



5, NC: not changeable. For all not changeable h ^ {0,-l} 



2 Each difference number will fall into one and only one of the above sets. 



4 The next step is to create a location map of all expanded (after embedding) 

5 difference numbers as indicated by block 306 in Figure 3. We partition the set 

6 EN into two disjoint subset EN1 and EN2. Every h in EN1 , will be expanded; 

7 and every h in EN2, will not be expanded (though it is expandable). A 

8 discussion on how to select expandable h ^ {0,-l} for difference expansion is 

9 given below. We create a one-bit bitmap, with its size equal to the numbers of 

1 0 pairs of pixel values. For the difference number in either EZ or EN1 , we assign 

11 a value "1" in the bitmap; for the difference number in either NZ, EN2, CNE, or 

12 NC, we assign a value "0". Thus a value "1" will indicate an expanded 

13 difference number. The location map will be lossless compressed by a JBIG2 

14 compression or run-length coding. The compressed bit stream will be denoted 

15 as 1.. An end of message symbol is appended at the end of L. 



17 We collect original LSB values of difference numbers in EN2 and CNE. For 

18 each h in EN2 or CNE, LSB{h) will be collected into a bit stream C. An 

19 exception is when /?=1 or -2, nothing will be collected. 



21 With the location map L, the original LSB values C, and a payload P (which 

22 includes an authentication hash, for example, an SHA-256 hash), we combine 

23 them together into one binary bit stream B 



25 Assuming b is the next bit in S, depending on which set h belongs to, the 

26 embedding (by replacement) will be 



3 



16 



20 



24 



27 



EZor EN1: h = 2-h + b 



28 



. EN2orCNE: A= - 

2 
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1 
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4 



• NZ or NC: no change on the value of h, b is passed to the next h 

After all bits in B are embedded, we apply the inverse integer transform (2) 
to obtain the embedded image. 
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The bit stream B has a bit length of (1L1+1C1+|P1). Assume the total number 
of 1 and -2 in EN2 and CNE is N, as each expanded pair will give one extra 
bit. The total hiding capacity will be (|C1+A/+|EZ|+1EN1|). Accordingly, to 
have B successfully embedded, we must have: 



\L\ + \C\ + \P\<\C\+N+\EZ\ + \Em: 



(4) 



I.e., 



I £ I + 1 P |< iV+ 1 £Z I + 1 ENl I (5) 

Note that if the bit stream C is loss-lessly compressed before embedding, then 
Condition (4) becomes 

\L\+a\C\ + \F\<\C\ +N+ \EZ\ + \ ENl \ 
where a is the achieved compression rate, 0 < a < 1 . 

The partition of expandable h ^ {o -l} into ENl and EN2 will be subject to 
Condition (5). We will give two designs, one for mean square error (MSE) 
consideration, and the other for visual quality consideration. 

Assume after difference expansion, an expanded pair (x, y) becomes (x', 
with the average number unchanged. 



= 2 



2-h + b 



hi 
2 
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1 Thus to minimize the mean square error, one should select h with small 

2 magnitudes for difference expansion. For example, one can pick a threshold T, 

3 and partition EN into EN1 and EN2 by checking whether the magnitude of h is 

4 less than or greater than T. 

5 

6 For the visual quality consideration, one can define a hiding ability of an 

7 expandable difference number, as follows. 

8 Definition For an expandable difference number h, if k is the largest number 

9 such that: 



12 The hiding ability tells us how many bits could be embedded into the difference 

13 number h without causing overflow and underflow. Thus for an expandable 

14 difference number /?, it will be at least log2 2 = 1, since k > 2. The hiding ability 

15 could be used as a guide on selecting expandable difference numbers. In 

16 general, selecting an expandable difference number with large hiding ability will 

17 degrade less on the visual quality than an expandable difference number with 

18 small hiding ability. A large hiding ability implies that the average of two pixel 

19 values is close to mid tone, while their difference is close to zero. 

20 For decoding, we do the pairing using the same pattern as in the embedding, 

21 and apply the integer transform (1) to each pair. Next we create two disjoint 

22 sets of difference numbers, C, and NC: 

23 1. C: changeable. For all changeable h 

24 2. NC: not changeable. For all not changeable h 



10 



k^h + h< min(2(255 - l\2l + 1) 



11 



forallO<b<.k-l, then we say the hiding ability of h is loga k. 
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26 



Then we collect all LSBs of difference numbers in C and form a binary bit 
stream S. From S, we first decode the location map. With the location 
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1 map, we restore the original values of difference numbers as follows 

2 (assuming b is the next bit from S): 



3 • if /? G C , the location map value is 1 , then h = 

4 next h 



, b is passed to the 



5 • if G C , the location map value is 0, and 0<h<l, then /7=1 , b is passed 

6 to the next h 

7 • if /z G C , the location map value is 0, and - 2 < /z < -1 , then h~2, b is 

8 passed to the next h 

9 • if /i e C , the location map value is 0, and h>2 or h<-3 , then 

h 



10 h 



2 + 



11 • \i h^C , the location map value should be 0 (otherwise a decoding error 

12 on a tampered image), no change on /?, b is passed to the next h 

13 After all difference numbers have been restored, we apply the inverse Integer 

14 transform (2) to reconstruct a restored image. If the embedded image has not 

15 been tampered, then the restored image will be identical to the original image. 

16 To authenticate the content of the embedded image, we extract the embedded 

17 payload Pfrom S, and compare the authentication hash in Pwith the hash of 

18 the restored image. If they match exactly, then the image content is authentic, 

19 and the restored image will be exactly the same as the original Image. Most 

20 likely, a tampered image will not go through to this step because some 

21 decoding error could happen in restoring difference numbers. This decoding 

22 error indicates that the image has been tampered. 

23 

24 The above described embodiment provides a high capacity, high quality, 

25 reversible watermarking method. The method partitions an image into pairs of 

26 pixel values (block 302 in Figure 3), selects expandable difference numbers for 

27 difference expansion (block 305 in Figure 3) and embeds a payload that 

28 includes authentication data (e.g., block 308 in Figure 3). By exploring the 



31 



wo 03/055130 



PCT/US02/40162 



1 redundancy in the image, reversibility is achieved. As difference expansion 

2 brings extra storage space, connpression is not necessary. Of course, 

3 employing compression can either increase the hiding capacity or reduce the 

4 visual quality degradation of watermarked image. 

5 

6 Detail of Second Embodiment: T he following is a detailed explanation of a 

7 second embodiment of the invention. This embodiment involves a reversible 

8 data embedding method for digital images. However, the method can be 

9 applied to digital audio and video as well. This embodiment is an example of 

1 0 expansion using N states for auxiliary data values to be embedded, where the 

1 1 state N corresponds to the level number L. 
12 

13 In this embodiment, two mathematical techniques are utilized, namely, 

14 difference expansion and Generalized Least Significant Bit (G-LSB) 

15 embedding. This embodiment achieves a very high embedding capacity, while 

16 keeping the distortion low. 
17 

18 In this embodiment, as in the first embodiment, the differences of neighboring 

19 pixel values are calculated, and some difference numbers are selected for 

20 difference expansion. The original G-LSBs values of the difference numbers, 

21 the location of expanded difference numbers, and a payload (which includes an 

22 authentication hash of the original image) may all be embedded into the 

23 difference numbers as indicated by bloc 308 in Figure 3. The needed extra 

24 storage space is obtained by difference expansion. With this embodiment, no 

25 compression is used. 
26 

27 This embodiment relates to watermarking a grayscale image. For color 

28 images, one can embed the data into each color component individually. 

29 Alternatively one can de-correlate the dependence among different color 

30 components, and then embed the data into the de-correlated components. 
31 
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2 
3 
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5 
6 
7 
8 

9 

10 

11 

12 
13 

14 

15 
16 

17 

18 

19 
20 

21, 

22 
23 

24 



The overall operation is as follows: a payload is embedded in a digital image I, 
to create an embedded image 1'. An image 1" is retrieved from the embedded 
image 1'. The retrieved image I" is identical to the original Image I, pixel by 
pixel, bit by bit. One can determine If the image 1' was tampered with by some 
intentional or unintentional attack using a content authenticator. The 
authenticator compares a hash of the original image I to a hash of the retrieved 
image I". 

This embodiment uses a reversible integer transform. 

The image being watermarked comprises grayscale-valued pairs (x, y). 

Each X and y has a value from 0 to 255. 

that is X, y e Z, 0 ^ x, y < 255. 
The average value 7 " and difference value "h" of the pairs is defined 



x + y 



h-x-y 



(21) 



where the symbol L J is the floor function meaning "the greatest integer less 
than or equal to". The inverse transform of equation 1 is: 



X = l + 



h 
2 



(22) 



In some of the literature, the reversible transform given in equations 21 and 22 
above is called the Haar wavelet transform or the S transform. 

The magnitude of the difference number h is used for embedding. Since 
grayscale values are in the range of 0 to 255, 



0 < / + 



< 255 ,0 < / - 



h_ 
2 



< 255 
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1 which is equivalent to: 

2 \h\ < min( 2(255 - l\2l + 1) (23) 

3 Thus to prevent overflow and underflow problems, the difference number h 

4 (after embedding) satisfies Condition (23). 



6 
7 



8 
9 

10 

11 

12 

13 

14 



15 
16 

17 

18 

19 

20 

21 

22 
23 
24 
25 



Given an integer L, L e, L > 2. the (L-level) G-LSB , g, of a difference number 
h, is the remainder of its magnitude after dividing by L, 



L 



The G-LSB g is the selected embedding area for this embodiment. In order to 
prevent any overflow and underflow problems during embedding, embedding 
only takes place in the changeable difference numbers defined as follows: 



For a grayscale-valued pair (x,y), the difference number h is L-chanoeable if : 

Ir + 1 < tmn(%(2m - + 1) 



During data embedding, the G-LSB g might be replaced by a value from the 

remainder set {0,1 , L-1}. In view of constraint set out in equation 23 above, 

some large remainders might cause an overflow or an underflow. Thus we 
replace g with a value from the partial remainder set {0,1 ..M}, with 

\h\ 

g < M < L-1 , where M is determined by: / and 



It is noted that modifying G-LSBs of L-changeable h (without compression) 
does not provide extra storage space. With this embodiment, extra storage 
space is gained from the expandable difference numbers. 
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1 In this embodiment, for a particular grayscale pair (x,y), a difference number h 

2 is called L-expandable if : 



3 
4 



l^h £ + 1 < mm(i(255 0. + 1) 



5 In a base L representation , an L -expandable h can add one extra number b 

6 after its G-LSB. More precisely, h could be replaced by a new difference 

7 number h\ without causing an overflow or underflow where h' is defined by: 

^ Big^{h) - {\h\ * JD + 6) 

8 

9 Again, due to the constraint in equation 23 above, b could be a value from a 
1 0 partial remainder set {0,1 , M) with 1< M < L -1 and M is determined by / 

1^1 



11 and 



. Thus, for each L-expandable difference number, one could gain 



12 log2(M+1) extra bits. The reversible operation h to /?' is termed "difference 

13 expansion". An L -expandable h is also L -changeable. After difference 

14 expansion, the expanded /?' is still L -changeable. 
15 

16 For h < 0, we can alternatively define L -changeable (and L -expandable) as: 

^ ^ Ir + 1 < miii(2(2iS l),2i + 1) 

17 ^ ^ 
18 

19 The Embedding Algorithm: A watermark is embedded in an image using the 

20 above described technique using the following procedure. First, The image is 

21 partitioned into pairs of pixel values as indicated by block 302 in Figure 3. A 

22 pair of pixels comprises two neighboring pixel values or two pixels with a small 

23 difference number as indicated in Figure 2A. The pairing could be through ail 

24 pixels of the image or just a portion of it. The integer transform (equation 21) is 

25 applied to each pair. 
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1 

2 In order to achieve maximum embedding capacity, one can embed a payload 

3 with one pairing, then embed another payload with another pairing on the 

4 embedded image. For example, we could embed column wise first, then 

5 embed row wise. 
6 

7 After applying the integer transform (equation 21) to each pair, five sets of 

8 difference numbers designated EZ, NZ, EN, CNE, and NC are created using 

9 the above definitions of changeable and L-expandable as follows: 
10 

11 1 . EZ: expandable zeros. For all L -expandable where h = 0 . 

12 2. NZ: not expandable zeros. For all not L -expandable where h = 0 . 

13 3. EN: expandable non zeros. For all L -expandable h^O. 

14 4. CNE: changeable, but not expandable. For all L - changeable, but not L - 

1 5 expandable h 0. 

16 5. NC: not changeable. For all not L -changeable h^O. 
17 

1 8 Each difference number will fall into one and only one of the above sets. 
19 

20 The next step (block 306 in Figure 3) is to create a location map of all 

21 expanded (after embedding) difference numbers. The set EN is partitioned into 

22 two disjoint subset EN1 and EN2. Every h in EN1 , will be expanded; every h in 

23 EN2, will not be expanded. (It is noted that to achieve maximum embedding 

24 capacity, EN1 would include the whole set EN, and EN2 will be empty). 
25 

26 A one-bit bitmap is created. Its size is equal to the numbers of pairs of pixel 

27 values (block 302 in Figure 3). For an h in either EN1 or EZ, a value 1 is 

28 assigned in the bitmap; otherwise a 0 is assigned. Thus, a value 1 indicates an 

29 expanded difference number. The location map is then loss less compressed 

30 by a JBIG2 compression or by run length coding. The compressed bit stream is 

31 denoted as U. An end of message symbol is appended at the end of L'. 
32 
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10 



1 Next, we collect the original values of G-LSBs of the difference numbers in EN2 

2 and ONE, For each h in EN2 or CNE, its G-LSB g is collected into a bit stream 

3 C. We employ a conventional L-ary to Binary conversion method to convert g 

4 to a binary bit stream. 
5 

6 The L-ary to Binary conversion is a division scheme of unit interval, similar to 

7 arithmetic coding. Since h is L -changeable, we determine M , where g could 

8 be replaced by a value from {0, 1, ... M } without causing an overflow or 

9 underflow. We convert g to the interval: 

g ^ g + 1 ^ 

1 1 The interval is further refined by the next G-LSBs, and so on, until we reach 

12 the last G-LSB. Then we decode the finalinterval to a binary bit stream. By 

1 3 using L-ary to Binary conversion, instead of simply using a fixed length binary 

14 representation of g, the representation of G-LSBs is more compact, which 

1 5 results in a smaller bit stream size of C. 
16 

17 It is noted that when L = 2, as M will always be 1 , there will be no need for the 

1 8 L-ary to Binary conversion. It is also noted that if Ihl ^ L - 1, after its g is 

19 collected, we also store its sign, sign(h ), in the bit stream C. 
20 

21 Finally, (as indicated by block 308 in Figure 3) we embed the location map U, 

22 the original values of G-LSBs C, and a payload P (which includes an 

23 authentication hash, for example, an SHA-256 hash). We combine them 

24 together into one binary bit stream S, 

25 S= U u C u P 

26 We use the inverse L-ary to Binary conversion to convert the binary bit stream 

27 S to M-ary, with M determined for each expandable difference number in EZ 

28 and EN1, and each changeable difference number in EN2 and CNE. The 

29 embedding (by replacement) is: 
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4 



2 



3 



1 



• EZ: Ihl = b, where b is the M -ary symbol from the Inverse L-ary to Binary 
conversion, and the sign of h Is assigned pseudo randomly. 

. EN1 : /7 = slgn( h ) • ( Ihl • L + ib ; . 

• EN2 or ONE: /7 = sign( h ) • ( -L + b). 



5 



• NZ or NC: no change on the value of h . 



6 After all embedding is done, we apply the inverse integer transform (equation 

7 22) to obtain the embedded image. 
8 

9 The Decoding Algorithm: The decoding process uses the same principles as 

10 the embedding process. First, we do pairing of pixels using the same pattern 

11 as in the embedding as indicated by block 401 in Figure 4. The integer 

12 transform (equation 21) is applied to each pair. 
13 

14 Next two disjoint sets of difference numbers, C, and NC are created as follows: 

15 1 . C: changeable. For all L -changeable h . 

16 2. NC: not changeable. For ail not L -changeable h . 

17 Next we collect all G-LSBs of difference numbers in C. We employ the L-ary to 

1 8 Binary conversion to convert it into a binary bit stream B. From the binary bit 

19 stream, we first decode the location map. With the location map, we restore 

20 the original values of difference numbers as follows: 

21 a) \fh€ C, and the location map value is 1 , then 

1^1 

22 h = sign {h) . ^ 



L 



23 



25 



24 



b) if h e C, and the location map value is 0, and h = 0, decode an M -ary 
symbol b from B, and decode a sign value s from B, then h = s» b . 



26 



c) if h e C, the location map value Is 0, and 1 < Ihl < L - ■/, 



27 



28 



then h = sign(h) . b , and the next sign value from B should correctly 
match sign(h ). 
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1 



2 



(i) '\f h € C, the location map value is 0, and Ihl > L , 



3 



4 



5 



e) \fhe NC, the location map value should be 0, no change on the value 
of/?. 



6 



7 After all difference numbers have been restored, we apply the inverse integer 

8 transform (equation 22) to reconstruct a restored image. If the embedded 

9 image has not been tampered, then the restored image will be identical to the 

10 original image. To authenticate the content of the embedded image, we extract 

1 1 the embedded payload Pfrom B. The authentication hash in P is compared 

1 2 with the hash of the restored image. If they match exactly, then the image 

1 3 content is authentic, and the restored image will be exactly the same as the 

14 original image. (Most likely a tampered Image would not go through to this step 

15 because some decoding error could happen before this step indicating a 

16 tampered image.) 



18 For the maximum embedding capacity all expandable difference numbers (EN1 

19 = EN) are expanded and the location map is loss less compressed by JBIG2. 

20 For more capacity and for other reasons, one can first embed with the column 

21 wise pairing, then embed with the row wise pairing on the column wisely 

22 embedded image. 



24 To embed a payload with a smaller size than the maximum embedding 

25 capacity, one can reduce the size of EN1, until the targeted embedding 

26 capacity is met. For example, to embed a payload of 138856 bits in a particular 

27 image in which there are 1 16029 expandable non-zeros at L = 2 with column 

28 wise pairing. One can assign 106635 of them in EN1, and the rest in EN2. 



17 



23 
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1 The PSNR of the embedded image is then higher than some other methods 

2 with a payload of the same size. 

3 The above described embodiment provides a high capacity reversible data 

4 embedding algorithm. The difference expansion provides extra storage space, 

5 and compression on original values of the embedding area is not needed. With 

6 compression (such as a linear prediction and entropy coding), the maximum 

7 embedding capacity will be even higher, at the expanse of complexity. 
8 

9 Third Embodiment: T he third embodiment uses the same reversible integer 

10 transform as used in the first and second embodiment and which is given by 

1 1 equations 1,21,2 and 22 above. Furthermore to prevent overflow and 

1 2 underflow conditions: 



13 0</ + 



<255, and 0</- 



<255 



14 since / and h are integers, the above is equivalent to: 

15 |/i'|< 2(255 -/X and \h\<2l + l (33) 

16 Condition (33) sets a limit on the magnitude (absolute value) of the difference 

17 number h. As long as h is in such range, it is guaranteed that (x, y) computed 

1 8 from Equation 2 or 22 will be a grayscale value. Condition given by equations 

19 33 above are equivalent to: 



20 \h]< 2(255 -IX if 128 </< 255 

21 \h\<21 + h if 0 </< 127 

22 For this embodiment Expandable and Changeable difference numbers are 

23 defined as follows: When a bit b is embedded into a difference number h by 

24 difference expansion, the new difference number h' is: 

25 h' = 2xh^b 

26 In accordance with equation 33 above, in order to prevent overflow and 

27 underflow, h' must satisfy the following conditions. 
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1 \h\ < mm(2(255 - l\2l 1) 

2 Definition of Expandable Difference number : for a grayscale-valued pair (x,y), 

3 which are members of a set Z and where Q<x,y<. 255, we define the average 

4 and difference: 



20 



22 



24 



x + y 



,h^x-y as previously explained 



6 The difference number h is expandable under / for both b= 0 and 1 if : 

7 [2 X /i + 6| < Mm(2(255 ~ l),2l + 1) 

8 It is noted that since an expansion does not change the average number / , so 

9 for simplicity and brevity, we say h is expandable, as an abbreviation of saying 
10 Aj is expandable under / . 

11 

12 For an expandable difference number /? , if we embed a bit by difference 

13 expansion, the new difference number li still satisfied conditions 33. so the 

14 new pair computed from equation 2 above is guaranteed to be a grayscale 

15 value. Thus expandable difference numbers are candidates for difference 

16 expansion. 

17 

18 As each integer can be represented by the sum of a multiple of 2, and its LSB 

19 (least significant bit), for new, expanded difference number h': 



/2' = 2X 



21 Ifwe modify its LSB; 



g = 2x 



2 



+ LSB{h ) with LSB(/7') = 0 or 1 . 



23 with jb' = 0 or 1 , then 



2x 




+ b' 




2x 


2xh + b 


+ b' 








_2_ 








L 2 J 
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1 =-\2xh + b'\< min(2(255 - /),2/ + 1) 

2 Thus after difference expansion, the new difference number h' could have its 

3 LSB modified, without causing an overflow or underflow. We call such a 

4 difference number changeable. 
5 

6 Definition of Changeable difference number: for a grayscale-valued pair (x,y), 

7 which are members of a set Z and where 0 < x,;; < 255, we define the average 

8 / and difference h as: 



9 / = 



11 



x + y 
2 



h^x-y as previously explained 



10 In this embodiment, the difference number h id defined as changeable if: 



2x 



+ 6 



< min(2(255 - /),2/ + 1) for both = 0 and 1/ 



1 2 From the above it follows that : 

13 1) If a difference number h is a positive odd number or a negative even 

14 number, it is always changeable. 

15 2) For a changeable difference number, after its LSB is modified, it is still 

16 changeable. 

17 3) An expandable difference number h is always changeable. 

18 4) After difference expansion, the new difference number h' is changeable. 

19 5) If /? = 0 or -1 , the conditions on expandable and changeable are equivalent. 
20 

21 The Location Map: One can select some expandable difference numbers, and 

22 embed one bit into each of them. However to extract the embedded data and 

23 restore the original grayscale values, the decoder needs to know which 

24 difference numbers has been selected for difference expansion. To facilitate 

25 identification of expanded values, we can embed such location information, 

26 such that the decoder could access and employ it for decoding. For this 

27 purpose, we create and embed a location map, which includes the location 

28 information of all selected expandable difference numbers. 
29 
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1 The data embedding Algorithm: The location map allows the encoder and the 

2 decoder to share the same information concerning which difference numbers 

3 have been selected for difference expansion. While it is straightforward for the 

4 encoder, the decoder needs to know where (from which difference numbers) to 

5 collect and decode the location map. 
6 

7 After difference expansion, the new difference number h' might not be 

8 expandable. On the decoder side, to check whether h' is expandable does not 

9 tell whether the original h has been selected for difference expansion during 

10 embedding. As we know, the new difference number h' is changeable, so the 

1 1 decoder could examine each changeable difference number. With the 

12 technique described here, the encoder selects changeable difference numbers 

13 as the embedding area. The decoder uses the same data to decode. During 

14 data embedding, all changeable difference numbers are changed, by either 

15 adding a new LSB (via difference expansion) or modifying its LSB. To 

16 guarantee an exact recovery of the original image, we will embed the original 

17 values of those modified LSBs. 
18 

19 In brief, data embedding algorithm used by this embodiment includes six steps: 

20 calculating the difference numbers, partitioning difference numbers into four 

21 sets, creating a location map, collecting original LSB values, data embedding 

22 by expansion, and finally an inverse integer transform. Each of these steps is 

23 discussed below. 
24 

25 The original image is grouped into pairs of pixel values. A pair comprises two 

26 neighboring pixel values or two with a small difference number. The pairing 

27 could be done horizontally by pairing the pixels on the same row and 

28 consecutive columns; or vertically on the same column and consecutive rows; 

29 or by a key-based specific pattern. For example, Fig. 2A show a pairing pattern 

30 that could be utilized. The pairing could be through all pixels of the image or 

31 just a portion of it 
32 
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1 The integer transform (equation 1 above) is applied to each pair. Then we 

2 design a scanning order for all the difference numbers h , and order them as a 

3 one dimensional \ist {hi, h2, Iim}' 

4 

5 Next, four disjoint sets of difference numbers are created, namely 

6 EZ, EN. CNE, and NC: 
7 

8 1) EZ: expandable zeros (and minus ones). For all expandable /? = 0 and 

9 expandable /j = -1. 

10 2) EN: expandable non-zeros. For all expandable h that are not a member of 

11 the set {0,-1} 

12 3) CNE: changeable, but not expandable. For all changeable, but non- 

1 3 expandable h. 

14 4) NC: not changeable. For all non-changeable h . 
15 

16 Each difference number will fall into one and only one of the above four sets. 

17 Since an expandable difference number is always changeable, the whole set of 

1 8 expandable difference numbers is EZ u EN, and the whole set of changeable 

19 difference numbers is EZ u EN u CNE. 
20 

21 The third step is to create a location map of selected expandable difference 

22 numbers. For a difference number h in EZ, it will always be selected for 

23 difference expansion. For EN, we partition it into two disjoint subset EN1 and 

24 EN2. For every h in EN1 , it will be selected for difference expansion; for every 

25 h in EN2, it will not (though it is expandable). A discussion on how to partition 

26 EN is given below. A one-bit bitmap is created vas the location map, with its 

27 size equal to the numbers of pairs of pixel values (in Step 1). For example, if 

28 we use horizontal pairing through all pixels, the location map will have the 

29 same height as the image, and half the width. For an h in either EZ or EN1 , we 

30 assign a value 1 in the location map; for an h in EN2, CNE, or NC, we assign a 

31 value 0. Thus a value 1 will indicate a selected expandable difference number. 

32 The location map will be lossless compressed by a JBIG2 compression or run- 
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1 length coding. The compressed bit stream is denoted as L . An end of 

2 message symbol is at the end of L , 
3 

4 In the fourth step, the original LSB values of difference numbers are collected 

5 in EN2 and CNE. For each h in EN2 or CNE, LSB(A? ) will be collected into a bit 

6 stream C . An exception is when /? = 1 or -2, nothing will be collected, as its 

7 original LSB value (1 and 0, respectively) could be determined by the location 

8 map information, (see the decoding section below for an explanation). 
9 

10 Fifth, we embed the location map L , the original LSB values C , and a payload 

1 1 . The payload P includes an authentication hash (for example, a 256 bits SHA- 

12 256 hash). The payload size (bit length) is limited by the embedding capacity 

13 limit discussed below. We combine L , C , and P together into one binary bit 

14 stream B, 

15 B-LuCuP- bi,b2 ^^^-b^ 

1 6 where: b. g {0,1}, 1 < i < m, m is the bit length of S. We append C to the end of L 

17 and append P to the end of C. The bit stream B is embedded into the 

1 8 difference numbers as follows. 

1) Scti- land j =-0. 

2) While (i < m) 

hj — 2 X hj " I- fef. 
i ^ i -|~ 1. 
. Elseif £ EN2 or hj e CNB 

19 3) End 
20 

21 
22 

23 Only changeable difference numbers (set EZ u EN u CNE) are modified, non- 
24 changeable difference numbers and all average numbers are unchanged. For 

25 a changeable difference number, either a new LSB is embedded by difference 

26 expansion (if it is in EZ or EN1) or its original LSB is replaced (if it is in EN2 or 
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1 CNE). Thus after embedding, all the embedded information are in the LSBs of 

2 changeable difference numbers. By collecting the LSBs of changeable 

3 difference numbers, the decoder will be able to recover the embedded bit 

4 stream 6 
5 

6 Finally after all the bits in B are embedded, the inverse Integer transform 

7 (equation 2 above) is applied to obtain the embedded (watermarked) image. 
8 

9 Capacity Limit: The bit stream B has a bit length of + |C| + \P\ where |.| is the 

10 cardinality (bit length or numbers of elements) of a set. The total embedding 

1 1 ■ capacity is (\EZ\ + \Em\ + \EN2\ + \CNE)\ . 
12 

1 3 For successful embedding we must have: 

14 |Z,| + |C| + |P| < \EZ\ + \EN1\ + \EN2\ + \CNE\ 

1 5 Assume the total number of 1 and -2 in EN2 and CNE is N, then 

16 jPl^l^lZj + lMl + iV'-lil (35) 

17 The payload size is upper bounded by the sum of the number of selected 

1 8 expandable difference numbers and the number of not selected or not 

19 expandable h e {1,-2}, minus the bit length of the location map. 
20 

21 Difference Number Selection: Due to the redundancy in pixel values of natural 

22 images, the difference numbers of neighboring pixel values are usually small. 

23 For a pair of two pixel values, if their integer average is in the range of [30, 

24 225], and their difference number is in the range of [ -29, 29] , then: 
25 

\2xh+b\ < 2 X \h\ + l^l < 2 X 29 + 1 
2g = 59 < 60 < inm(2(255 - I), 21 4- 1), 

27 

28 for both b =0 and 1 , and the difference number h is expandable. 
29 
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1 Since most integer averages and difference numbers will be in such ranges, 

2 most difference numbers will be expandable. We have found that, in general, 

3 many natural grayscale images usually have over 99% expandable difference 

4 numbers. If all expandable difference numbers are selected for difference 

5 expansion, the location map is very compressible (as over 99% values are 1), 

6 the embedding capacity limit will be close to 0.5 bpp. When the payload has a 

7 bit length less than the capacity limit, we only need to select some expandable 

8 difference numbers for difference expansion. 
9 

10 With a given payload P , the selection of expandable difference numbers in EN 

1 1 for difference expansion is constrained by condition (35) above. We present 

12 two simple selection methods here, one for mean square error (MSE) 

13 consideration, and the other for visual quality consideration. 



15 For a grayscale-valued pair ( x, y) , assume the new grayscale valued pair after 

16 difference expansion is ( x' , / ) . Since the average number / is unchanged, 

17 and we have: 
18 



21 Thus the Euclidean distance between the original pair ( x, y ) and the new, 

22 expanded pair ( x', y' ) is proportional to the difference number h (before 

23 difference expansion). To minimize the MSE between the original image and 

24 the embedded image, we should select h with small magnitudes for difference 

25 expansion. We choose a threshold T, and partition EN into EN1 and EN2 by 



2x (a/ 



19 
20 
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mi = {h£ EN : \h\ < T], EN2 = {h G EN : \h\ > T}. 

1 

2 For a payload P, we start with a small threshold 7, then increase 7 gradually 

3 until Condition (35) above is met. One could preprocess an image and create a 

4 threshold vs. capacity limit table, by calculating (lEZl + IEN1I + N -ILI). When 

5 proceeding to embed a payload, one could check this table and pick an 

6 appropriate threshold. 
7 

8 For the visual quality consideration, we can define a hiding ability of an 

9 expandable difference number, as follows. If k is the largest integer such that: 

\k X + li| < iiiiii(2(255 - 1), 21 + 1), 

10 

11 for all 0 < 6 < A: - 1 , we can say the hiding ability of h is loga k. 
12 

1 3 For a difference number h with hiding ability log 2 /c , we can replace h with a 

14 new difference number kxh + b , where b g {0 k - 1}, without causing an 

1 5 overflow or underflow. This means we could reversibly embed log 2 k bits. For 

16 an expandable difference number, as /c will be at least 2, its hiding ability will be 

1 7 at least log 2 2 = 1 . Although with this embodiment we do not embed more than 

18 one bit into a difference number, the hiding ability could be used as a guide on 

1 9 selecting expandable difference numbers for difference expansion. 

20 

21 In general, selecting an expandable difference number with large hiding ability 

22 will degrade less on the visual quality than an expandable difference number 

23 with small hiding ability. A large hiding ability implies that the average of two 

24 pixel values is close to mid tone, while their difference is close to zero. Again 

25 we can choose a threshold T , and partition EN into EN1 and EN2 by : 
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ENl - {/i e EN : mdingAhility(;i) > T}, 
EN2 - {/i € EN : ffidingAbimy(/i) < T}. 

1 

2 It should be noted that with a different threshold T in the above two selection 

3 methods, the location map L also changes, so does its bit length C. Thus a 

4 third method to partition EN could be based on the compressibility of the 

5 location map. We could select expandable difference numbers such that the 

6 location map is more compressible by lossless compression. 
7 

8 JBIG2 Compression ; The location map (before loseless compression) is a one- 

9 bit bitmap. It can be efficiently compressed by JBIG2, the new international 

10 standard for lossless compression of bi-level images. JBIG2 supports model- 

1 1 based coding to permit compression ratios up to three times those of previous 

12 standards for lossless compression. For more details on JBIG2, we refer to an 

13 article by P.G. Howard, F. Kossentini, B. Martins, S. Forchammer, and W.J. 

14 Rucklidge, "The emerging JBIG2 standard" IEEE Transactions on Circuits and 

15 systems for Video Teclinology, vol. 8, no. 7 pp 838-848, 1998. For our 

16 reversible data embedding method, we can employ a slightly modified and 

17 more compact JBIG2 encoder and decoder, as we can discard most of the 

1 8 header information in the standard JBIG2 bit stream. 
19 

20 It should be noted that the last two bytes of the JBIG2 bit stream are the end of 

21 message symbol. The second to last byte will always be 255, and the last byte 

22 will be greater than 143 (it is 173 in a JBIG2 bit stream from Power JBIG-2 

23 encoder developed by the University of British Columbia). With the end of 

24 message symbol, our decoder can separate the location map C from the next 

25 bit stream C easily. 
26 

27 Multiple Embedding: It is possible to employ the technique described here to 

28 an image more than once for multiple embedding. For an already embedded 

29 image, we can embed it again with another payload. Even for one payload, we 
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1 can divide the payload into several pieces and use multiple embedding to 

2 embed them. As we have a choice of pairing of pixel values in Step 1 during 

3 embedding, we can use a different pairing for each embedding. One approach 

4 is to use a complement pairing. For example, if the image is embedded with a 

5 horizontal pairing, then we can use a vertical pairing for the next embedding. 

6 Other approaches are also possible. As each embedding has an embedding 

7 capacity limit less than 0.5 bpp, a multiple embedding will have an embedding 

8 capacity limit less than M/ 2 bpp, where M is the number of embedding. 
9 

10 In order to assist the decoder to determine whether or not there has been 

1 1 multiple embedding, one can embed header information before the location 

12 map G. The bit stream B now becomes; 

13 
14 

15 where // is a 16 bit header. For the original image (first embedding), H is set 

16 to 0. The pairing pattern of the original image will be the H at the second 

17 embedding. The pairing pattern of the second embedding will be theH at the 

18 third embedding, and so on. For a 16 bit H we have 2^^ - 1 = 65535 different 

19 pairing patterns to choose from. 
20 

21 Security : For security, the bit stream B can be encrypted by the Advanced 

22 Encryption Standard (AES) algorithm prior to embedding. 
23 

24 Decoding and authentication : The LSBs of changeable difference numbers 

25 are collected from the bit stream B . By collecting LSBs of all changeable 

26 difference numbers, we can retrieve the bit stream B . From B , we can decode 

27 the location map L and the original LSBs values G. The location map gives the 

28 location information of all expanded difference numbers. For expanded 

29 difference numbers, an (integer) division by 2 will give back its original value; 

30 for other changeable difference numbers, we restore their original LSB values 
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1 from the bit stream C . After all changeable difference numbers have restored 

2 their original values, we can restore the original image exactly, as non- 

3 changeable difference numbers and all average numbers are unchanged 

4 during embedding. 
5 

6 The decoding and authentication process consists of five steps. First we 

7 calculate the difference numbers. For a (possibly) embedded (and possibly 

8 tampered) image, we do the pairing using the same pattern as in the 

9 embedding, and apply the integer transform (1) to each pair. We use the same 

10 scanning order to order all difference numbers as a one dimensional list {hi, h2, 

11 hMl 

12 

13 Next we create two disjoint sets of difference numbers, C, and NC: 

14 1) C: changeable. For all changeable h . 

15 2) NC: not changeable. For all non-changeable h . 

16 Note that we do not need to examine expandability at the decoder. 
17 

18 Third we collect all LSB values of difference numbers in C, and form a binary 

1 9 bit stream B = ib i d 2 ... " - m - 
20 

21 Fourth, we decode the location map from B by JBIG2 decoder. Since the 

22 JBIG2 bit stream has an end of message symbol at its end, the decoder knows 

23 exactly the location in B, where it is the last bit from the embedded location 

24 map bit stream L. 
25 

26 In this embodiment, we assume the first s bits in B are the location map bit 

27 stream L (including the end of message symbol). Thus the embedded original 

28 LSB values C starts from the (s +1)-th bit in 6. We restore the original values 

29 of difference numbers as follows. 
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1) Seti = s+L 

2) Forj=l:n 

If the location map value at hy is 1 

Else 

lf(0<hj<iy 

hj = 1. 
Elseif (-2 <h^< -1) 

Else 

-I 3) Bud 

2 If the location map value is 1, the difference number has been expanded during 

3 embedding. Conversely, for a non-changeable difference number, its location 

4 map value must be 0, otherwise the image has been tampered. 
5 

6 For a changeable difference number h, if its location map value is 0, then its 

7 original value will be differed from h by LSB. If 0 < < 1 , the original value of h 

8 must be 1 . The reason is that the original value could be only either 0 or 1 , as it 

9 is differed from h by LSB. If the original value of h was 0, then it would be an 

10 expandable zero (as changeable zero is expandable), and its location map 

1 1 value would be 1 , which contradicts the fact that the location map value is 0. 

12 Similarly if -2 :<h < -1 , the original value of h must be -2. For other 

13 changeable difference numbers, we restore their original LSB values from the 

14 embedded bit stream C. 
15 

16 The fifth and last step is content authentication and original content restoration. 

17 After all difference numbers have been restored to their original values, we 

18 apply the inverse integer transform (2) to reconstruct a restored image. To 

19 authenticate the content of the embedded image, we extract the embedded 

20 payload P from B (which will be the remaining after restoring difference 

21 numbers). We compare the authentication hash in P with the hash of the 

22 restored image. If they match exactly, then the image content is authentic, and 
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1 the restored image will be exactly the same as the original image. (Most likely a 

2 tampered image will not go through to this step because some decoding error 

3 could happen in Step 4, as a non-changeable difference number might have a 

4 location map value 1 or a syntax error in JBIG2 bit stream.) 
5 

6 The decoding and authentication process for this embodiment operates as 

7 follows: It reconstructs a restored image I" from the embedded image \\ then 

8 authenticates the content of V by comparing the hash of the restored image I" 

9 and the decoded hash in P. If V is authentic, then the restored image I" will be 
10 exactly the same as the original image I. 

11 

12 For multiple embedding, the first 16 bits in B is the pairing pattern H , After the 

13 first 16 bits are extracted, we decode the location map, reconstruct a restored 

14 image, and authenticate the content. If the content is authentic, we use H as 

15 the pairing pattern to decode the restored image again. The decoding process 

16 continues until H = 0 or until tampering has been discovered (either a hash 

17 mismatch, JBIG2 decoding error, or wrong location map value). If H = 0, and 

18 no tampering has been discovered during the whole decoding process, then 

19 the final restored image will be exactly the same as the original image, pixel by 

20 pixel, bit by bit. 
21 

22 Fourth Embodiment: This embodiment provides a reversible watermarking 

23 method of digital images. While the embodiment specifically applies the 

24 method to a digital image, the method can be applied to digital audio and 

25 video as well. This embodiment employs an integer wavelet transform to 

26 losslessly remove redundancy in a digital image to allocate space for 

27 watermark embedding. The embedding algorithm starts with a reversible 

28 color conversion transform. Then, it applies the integer wavelet transform to 

29 one (or more) de-correlated component(s). The purpose of both the reversible 

30 color conversion transform and the integer wavelet transform is to remove 

31 irregular redundancy in the digital Image, such that we can embed regular 

32 redundancy into the digital image, for the purpose of content authentication 
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and original content recovery. Tlie regular redundancy could be a hash of the 
image, a compressed bit stream of the image, or some other image content 
dependent watermark. In the integer wavelet domain, we look into the 
binary representation of each wavelet coefficient and embed an extra bit 
into an "expandable" wavelet coefficient. Besides original content retrieval 
bit streams, an SHA-256 hash of the original image will also be embedded 
for authentication purposes. The method used in this embodiment is based 
on an integer wavelet transfomi, JBIG2 compression, and arithmetic coding. 

The following is a simple example that illustrates the process. Assume that 
we have two grayscale values (x,y), where x,y e Z, 0< x,y :^ 255, and that 

we would like to embed one bit b with b e {0,1} into (x.y) in a reversible way. 
More specifically let us assume: 

x = 205, y = 200, and £> = 0 
First we compute the average / and difference h of and y: 



/ = 



x + y 



205 + 200 



= 202, ;2 = x-j; = 205-200 = 5 



It is noted that the symbol [ J demotes the integer part of a number. For 
Example: 

L2.7j = 2, L-1-2J--2 
Next we expand the difference number h into its binary representation: 
h=5=10l2 

Then we add b into the binary representation of h at the location right after the 
most significant bit (MSB). It is noted that the MSB is always 1 . 

h'= WOI2 = IOOI2 =9 
Finally we computer the new grayscale values, based on the new difference 
number /j'and the original average value number / : 



x' = l + 



h'+l 


= 202 + 


9 + 1 


2 




2 



= 207, y = x'-/z' = 207- 9 = 198 



54 



wo 03/055130 



PCTAJS02/40162 



= 202, h'= X' Y = 207-1 98 = 9 



1 From the embedded pair (x',y'), the watermark detector can extract the 

2 embedded bit b and get back the original pair (x,y) by a process similar to the 

3 embedding process. Again, we compute the average and difference: 

2 

5 The binary representation of h' is: 

6 A?'=9 = 10012 

7 Extracting the second most significant bit, which is "0", as the embedded bit b 

8 which leaves: h"= IOI2 =5 
9 

10 Now with the average r and difference h'\ we can retrieve exactly the original 

1 1 grayscale value pair (x,y). 
12 

13 In the above example, although the embedded pair (207,198) is still 8 bpp, we 

14 have embedded an extra bit by increasing the bit length of the difference 

15 number h from 3 bits (which is the number 5) to 4 bits (which is the number 9). 

16 Such an embedding process is totally reversible. 
17 

18 Stated in a general manner: If we have a sequence of pairs of grayscale values 

19 {x,,y,Xix2^y2y-^(x„,y„) where x^.y, ^Zfi<x,,y, <255J<i<n 

20 one can embed the pay load: 6 = {63 , &2 } where 6,. e {0,1} ,1 <i<n by 

21 repeating the above process, 

X, -i- y^ 



22 /, 



=Xj -y^,l<i<n. 



23 For each difference number h\ expand It to a binary representation: 

24 ~ ^/,0^U ^iJii) 

25 where n,© = 1 is the MSB, n.m e {0,1}, for 1 < m < j(i). with j(i) +1 as the bit 

26 length of /?, in its binary representation. Then we could embed bf into /?/ by 

27 ^/ ~ ^i,0^i^i,l'-^ij{i). 
28 
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1 Alternatively, we can combine all the bits e{0.1}, with 1 < m < j(i), 1 < I ^ 

2 n and b = [b].} into a single bit stream. Note, that we do not select the MSBs. 
3 

4 B = '"l,! '"l, 2 ^ 1,7(1)^2,1^2,2 ^2,j(2) •••• ^/?,l^/7,2 ^ n J {n)^ \^ 2 — ^ r. 

5 and use a reversible mapping / which could be encryption, loss-less 

6 compression, or other invertible operations or a combination of such operations 

7 to form a new bit stream C: 

8 C= f{B) = CiC2 Ck 

9 where q g {0,1}, for 1 < i < k, with k as the bit length of C. Then we could 
10 embed C into the difference numbers 1 :S i < n by 

12 where: 

13 ^^(/-i)-f2 ^s(j) is a truncated subsequence of C 

14 with: 

15 s(0) = 0, and = s(i-1) + j(i) + 1 

16 The length of h] is still one than that of /?,. For detection / is reversible, we 

17 can get back B by C) , 

18 and consequently, we can get back the original pairs (x^,yi),(x2,y2)'^'^^(x„,y„) 
19 

20 The reason we could increase the bit length of the difference number of an 

21 image is because of the high redundancy in pixels values of natural images. 

22 Thus, in most cases h will be very small and have a short bit length in its binary 

23 representation. In an edge area containing lots of activity, the difference 

24 number h from a pair of grayscale values could be large. For example if x = 

25 1 05, y = 22, the h = X - y = 83 =1 01 001 12 . If we embed a bit "0" into h, 

26 /?'=1001001l2 = 147. with / = 63 unchanged, the embedded pair will be x' = 

27 137, y' = -10. This will cause an underflow problem as grayscale values are 

28 restricted to the range [0,255]. Below we provide definition of "expandable 

29 pairs", which will prevent overflow and underflow problems. 
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1 

2 Reversible color conversion : The reversible color conversion transform 

3 discussed below de-correlates the dependence among different color 

4 components to a large extent, it is a loss-less color transform and the 

5 transform output is still integer-valued. For a RGB color image, the 

6 reversible color conversion transform is: 



7 



8 
9 

10 

1 1 Its inverse transform will be 



Ur = R-G, 
Vr = B-G. 



G = Yr 



R = Ur + G, 
B = Vr + G, 

12 
13 

14 The reversible color conversion transform maps a grayscale valued triplet to an 

1 5 integer triplet. It can be thought of as an integer approximation of the CCIR 

16 601 standard which provides a conversion to YcrCb space defined by the 

17 following matrix. 



(V \ ( 0.299 0.587 0.114 \ ( R\ 
Cr j = I 0.500 -0.419 -0.081 ) j G J . 
Ch } \ -0.169 -0.331 0.500 ) \ B ) 



18 

19 

20 The RGB to YCrCb transform matrix is not integer-valued. It requires floating 

21 point computing. Such a transform will introduce small round off errors, and will 

22 not be a reversible transform. Since reversible watermarking requires original 

23 retrieval with 1 00% accuracy, we use the reversible color conversion transform 

24 instead of the RGB to YcrCb transform. 
25 
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1 For a grayscale image there will be no reversible color conversion transform 

2 since we apply the integer wavelet transform directly. 
3 

4 Integer Wavelet Transform: T he integer wavelet transform maps integers to 

5 integers and allows for perfect invertibility with finite precision arithmetic (i.e. 

6 reversible). The wavelet filters for integer wavelet transforms are dyadic 

7 rational, i.e., integers or rational numbers whose denominators are powers of 2, 

8 like 13/4, -837/32. Thus the integer wavelet transform can be implemented 

9 with only three operations, addition, subtraction, and shift, on a digital 

10 computer. The fast multiplication-free Implementation is another advantage of 

1 1 the integer wavelet transform over standard discrete wavelet transform. 
12 

1 3 For example, for the Haar wavelet filter, the integer wavelet transform will be 

14 the average and difference calculation. 



16 And for a biorthogonal filter pair with four vanishing moments for all four filters, 

17 the integer wavelet transform will be: 



19 In this embodiment, we use will the Haar integer wavelet transform. The 

20 generalization to other integer wavelet transforms is understandable from this 

21 example. 

22 

23 After the reversible color conversion transform, we apply the integer wavelet 

24 transform to one (or more) de-correlated component. In this embodiment, we 

25 choose the Yr component, which is the luminance component. For a grayscale 



15 
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1 image, one can apply the integer wavelet transform directly to the whole image. 



2 
3 
4 
5 
6 
7 
8 
9 



Expandable Wavelet Coefficient: For the grayscale-valued pair (105, 22) and a 
payload bit "0" (or "1"), a brute-force embedding will cause an underflow 
problem. Now we will show how to prevent the overflow and underflow 
problems. 

For a grayscale-valued pair (x, y), where x, y s Z, 0 < x,y< 255, define the 
average and difference as: 



10 



he + y 
L 2 



11 
12 

13 
14 
15 



Then the inverse transform to get back {x, y) from the average number / and 
difference number h is: 



(41) 



Thus to prevent the overflow and underflow problems, i.e., to restrict x, y in the 
range of [0, 255] is equivalent to have: 



16 



17 



20 
21 



0<l + 



h + 1 



<2SS, 0<l- 



18 Since both / and h are integers, one can derive that the above inequalities are 

19 equivalent to: 



(42) 
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1 Condition (42) above sets a limit on the absolute value of the difference number 

2 h. As long as h Is in such range, it is guaranteed that (x, y) computed from Eqn. 

3 (41) will be grayscale values. Furthermore, Condition (42) is equivalent to 



/ 1^1 < 2(255 - 1) , if 128<l< 
\ |&|<2f + l, 0<l<12 



< I < 2S5 

4 
5 

6 With the above condition, we now define an expandable grayscale-valued pair. 

7 Definition: For a grayscale-valued pair (x, y), where x, y g Z, 0 < x, y < 

8 255, define 



9 
10 

11 

12 Then (x, y) is an expandable pair if and only if 

< niin (2{25& — I), 2i + 1) * 

"1 3 
14 

1 5 Note that if /? 9^ 0, the bit length of the binary representation of h is Llog2|/z|J+ 1. 



16 Thus 
17 



18 is the largest number whose bit length is one more than that of \h\ . Thus for an 

19 expandable pair (x, y), if we embed an extra bit ("0" or "1") into the binary 

20 representation of the difference number h at the location right after the MSB, 

21 the new difference number h' still satisfies Condition (42), that is, the new pair 

22 computed from Eqn. (41) is guaranteed to be grayscale values. For simplicity, 

23 we will also call h expandable if (x, y) is an expandable pair. 
24 
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1 Thus from the average number / , one can tell whether or not a difference 

2 number h is expandable, i.e., whether or not the bit length of h could be 

3 increased by 1 without causing any overflow or underflow problem. Further we 

4 define the changeable bits of h as: 
5 

6 Definition: For a grayscale-valued pair (x, y), assume h ^ 0, and the binary 

7 representation of 1^1 is: 



9 where: =l,r^ e{04}, for \<m< j\ wih y >0 and y+1 is the bit length. If g<y 
10 is the largest number: 



+ 2^ - 1 < min(2(255 - 1) , 21 + 1) , 



11 
12 

13 

14 then we say (x.y), or equivalently /?, has g changeable bits, and they are: 



15 

16 Since: 



i. 

18 by definition, h has g changeable bits if the last g bits in the binary 

19 representation are all changed to "1", it still satisfies Condition (42), or the new 

20 pair computed from Eqn. (41) is still grayscale values. Let's look at two 

21 extreme cases: 

22 If g = 0, then h has no changeable bits. 

23 If g = j, then all bits (excluding the MSB) in its binary representation are 

24 changeable. It is clear that if h is expandable, then g = / However the 

25 inverse is not true, i.e., g-j does not imply h is expandable. 
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1 The number "0" does not have a proper binary representation. We can 

2 increase it (along with all positive numbers) by 1 to fit it Into the definition of 

3 expandable and changeable. With such preparation, we extract bits from 

4 wavelet coefficients as follows: 

5 1 . For the Yr component of a color image or a grayscale image, apply 

6 the integer wavelet transform. 

7 2. If hi > 0 and < 255, we increase hi by 1 , A?/ = hi + 1 . 

8 3. Construct a bit stream R, which consists of changeable bits from all hi . 

9 The scanning order of hi is determined by a fixed pattern (for example, 
10 zigzag). 

11 

12 JBIG2 Compression: For a grayscale-valued pair (x, y), by the above 

13 definition we can tell whether or not it is expandable. When (x, y) has been 

14 modified by the embedder, it will not be clear to the watermark detector 

15 whether or not the original pair has been expanded, i.e., whether the bit length 

16 of the binary representation of the difference number has been increased by 1 

17 (thus larger than the original one), or it is the same as the original one. In 

18 order to remove the watermark and retrieve the original, un-watermarked 

19 image, the detector needs to know the location of expanded difference 

20 numbers h in the original image. 
21 

22 We can define a location map of expanded difference numbers by setting its 

23 value to "1" at each location when it is expanded or "0" otherwise. The 

24 location map can be viewed as a bi-level image. To store the location map, we 

25 can losslessly compress the bi-level image and store the compressed bit 

26 stream instead. We will employ JBIG2, the new international standard for 

27 lossless compression of bi-level images, to compress the location map of 

28 expanded difference numbers /?. For convenience, we will denote the JBIG2 

29 compressed bit stream of the location map of expanded h as J. Alternatively, 

30 the location map could be compressed by run-length coding. 
31 
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1 Arithmetic Coding: To make more room for embedding the payload, we can 

2 further iosslessly compress the collected bit stream R, which are all the 

3 changeable bits from difference numbers h. Either arithmetic coding or 

4 Huffman coding could be used for this purpose. In this embodiment, we use 

5 arithmetic coding 

6 C = ArithmeticCoding( R ; 

7 where C is the compressed bit stream from the arithmetic coding. 
8 

9 SHA-256 Hash: To authenticate a watermarked image and detect tampering, 

10 we embed a hash of the image into itself. The new hash algorithm SHA-256 is 

11 a 256-bit hash function that is intended to provide 128 bits of security against 

12 collision attacks. SHA-256 is more consistent with the new encryption 

13 standard, the Advanced Encryption Standard (AES) algorithm, than SHA-1 , 

14 which provides no more than 80 bits of security against collision attacks. We 

15 calculate the SHA-256 hash of the digital image (before the reversible color 

16 conversion transform) and denote the hash as H. 
17 

18 Embedding: W ith the compressed bit stream J of the location map, the 

19 compressed bit stream C of changeable bits, and the SHA-256 hash H (a 256 

20 bit stream), we are ready to embed all three of sets into changeable bits of 

21 difference numbers h in the integer wavelet domain. First we combine the sets 

22 into one big bit stream: 

23 

24 where 

25 4Ji€{0,l}, l<iS* 

26 and k is the bit length of S. 

27 As indicated above, we append C to the end of J, and append H to the end of 

28 C. The order of J, C, and H could be changed, as long as the embedder and 

29 the detector use the same order. Next we design a pseudo random scanning 

30 order for all the difference numbers h. This pseudo random order will be 

31 different from the scanning order used to construct the changeable bit stream 
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1 R. With the pseudo random order of h, we embed the bit stream S into h by 

2 replacing (part of) changeable bits. For expandable h, we increase the bit 

3 length of /? by 1 . thus increase the number of changeable bits by 1 . The 

4 following is a description of the embedding: 
5 

6 
7 

1. Assume ali diflferen^^ muribeta h Me ordered by the pseudo random order as fti, ftsii • • s hn^ 

% Set i = 1. 

3, Ifi<nand Jfe>0, 

^ If hi expandable, \hi\ » rori * - rj, and g ^ 

Set {All =i r^Ori « *rj^ novr \hi\ hm i + 1 dbtasi^able bits. 
Replace diaBgeable bife? in fcf with aib^^4*i7 ^ib*--igfHhS^9 * * * > ^ife- 
For m^l:g 

• JShi>Q, 

Set hi^hi — 1. 
Set f « i + X, & A — * 

g 4* Go to Step 3* 

9 

10 We modify only the absolute value of h, and keep the sign (and its MSB) 

1 1 unchanged. If h is non-negative, since it has been increased by 1, after bit 

1 2 replacement, positive h will have its value decreased by 1 . 
13 

14 The bit stream S is embedded by replacing changeable bits in difference 

15 numbers h. The capacity of all changeable bits will be much larger than the bit 

16 length of S. For example, the capacity of all changeable bits (including 

17 expanded bits) of a particular image could be 330,000 bits, while S is about 

18 210,000 bits. In such a case there could be about a 120,000 bits surplus, 

19 which is 0.45 bpp for an image size 512 x 512. This is a huge extra space 

20 which could embed additional information (such as a compressed bit stream of 

21 the image for locating tampering and recovery). So after embedding all bits in 

22 S, a large portion of changeable bits will not be changed. We can select 

23 changeable bits based on how much difference it will introduce (how much it 
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1 degrades the image quality) if it is changed during the embedding. We will 

2 discuss two difference cases here, non-expandable h and expandable h, 
3 

4 Modifying changeable bits in non-expandable h brings imperceptible changes 

5 to images. For example, in a sample image, if we restrict ourselves by not 

6 increasing the bit length of expandable h, and modify changeable bits only, 

7 then the worst possible distorted image is when we set changeable bits in h to 

8 be all equal to 1 or all equal to 0, depending on each h's value. In such a 

9 sample image, although the pixel value difference between the original and the 

10 distorted one is as large as 32, the visual difference between them is almost 

1 1 imperceptible. 
12 

13 For expandable h, if we increase its bit length by 1 and embed one more bit 

14 into it, the visual quality degradation could be very noticeable when \h\ is large, 

15 like in an edge area or an area containing lots of activity. To achieve best 

16 image quality, the extra changeable bits which are not used for embedding 

17 should be allocated to those expandable h with large absolute values. If \h\ is 

18 large, even if h is expandable, we can treat it as non-expandable by turning it 

19 off to "0" in the location map. 
20 

21 For security reasons, the compressed bit streams ,rand C from JBIG2 and 

22 arithmetic coding can be encrypted by the AES algorithm, before they are 

23 embedded into changeable bits of difference numbers h. 
24 

25 Authentication: with respect to changeable bits, if we assume h has g 

26 changeable bits, and its binary representation is: 



27 
28 



and if we arbitrarily change its changeable bits: 



29 



(45) 
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e {0.1}, 1 <i <g 

2 then the new pair defined by Eqn. (41) is still grayscale-valued, and the 

3 changeable bits of h' is exactly g. 
4 

5 Since the embedder does not change the average numbers / , the 

6 authenticator w\\\ derive exactly the same number of changeable bits in the 

7 difference number as the embedder. For expanded h whose bit length of its 

8 binary representation has been increased by 1 during the embedding, the 

9 authenticator will know such information from the location map. Thus, the 

10 authenticator knows exactly which bits have been replaced and which 

1 1 difference numbers are expanded (by one bit) during the embedding process. 

12 All these are crucial to retrieve back the original, un-watermarked image with 

13 100% accuracy. 
14 

15 The authentication algorithm is similar to the embedding algorithm. The 

16 authentication algorithm goes through a reversible color conversion transform 

17 and the integer wavelet transform. From wavelet coefficients, it extracts all 

18 changeable bits, ordered by the same pseudo random order of the 

19 embedding. From the first segment of extracted bits, it decompress the 

20 location map of expanded difference numbers h. From the second segment, it 

21 decompresses the original changeable bits values. The third segment will 

22 give the embedded hash. From equation (45) above, one knows which bits 

23 are modified and which bits are extra expanded bits during the embedding. 

24 Thus one can reconstruct an image by replacing changeable bits with 

25 decompressed changeable bits. The extracted hash and the SHA-256 hash of 

26 the reconstructed image can be compared. If they match bit by bit, then the 

27 watermarked image is authentic, and the reconstructed image is exactly the 

28 original, un-watermarked image. 
29 

30 In summary, this fourth embodiment provides a reversible watermarking 

31 method based upon the integer wavelet transform. The location map of 
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1 expanded wavelet coefficients, cliangeable bits of all coefficients, and an SHA- 

2 256 hash are embedded. An authenticator can remove the reversible 

3 watermark and retrieve an image, which is exactly the same as the original 

4 image, pixel by pixel. 
5 

6 While several specific embodiments have been described, those skilled in the 

7 art will realize that many alternative embodiments are possible using the 

8 principles described above. Furthermore the invention has a wide array of 

9 uses in additions to those discussed above. 
10 

1 1 For example, the present invention could be used to encode auxiliary data in 

12 software programs, manuals and other documentation. The technique could be 

13 used for the dual function of protecting the software (e.g., the software would 

14 not run until the embedded data was extracted with a secret key) and carrying 

15 auxiliary data related to the software, such as the manual or other program 

16 data. Alternatively, the software documentation may be embedded with 

17 executable software as the auxiliary data using the reversible embedding 

18 method. 
19 

20 A reversible watermarking scheme with two or more layers of embedded 

21 auxiliary data may be used to control the quality of distributed audio, video and 

22 still image content and control access to higher quality versions of that content. 

23 For example, a lower quality preview edition of the content can be embedded 

24 with one or more layers of reversible watermarks. As the user obtains rights to 

25 higher quality versions, the user can be provided with a key to reverse one or 

26 more layers of the reversible watermark, improving the quality of the content as 

27 each layer is removed. This approach has the advantage that the reversible 

28 watermark enables control of the quality, access to higher quality versions 

29 through reversal of the watermark, and additional metadata carrying capacity 

30 for information and executable instructions related to the content. 
31 
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1 A reversible watermarking scheme can also be used to distribute a key inside 

2 of content. For example, a preview sample version of the content could include 

3 decryption keys to decrypt other related content. 
4 

5 The technique can be applied to encrypted content, where the reversible 

6 watermark carries decryption keys that are extracted and then used to decrypt 

7 content once the watermark has been reversed. 
8 

9 As explained above, one has freedom to pick pairs as one desires. One could 

10 choose a location map that provides the redundancy in the values of each pair 

1 1 that provides for better embedding capacity. This might make the location map 

12 more complex, but it would be possible. 
13 

14 It is noted that watermarking software with the present invention would in effect 

15 "introduce reversible errors" into the software. Thus, the watermark prevents 

16 execution of the software by anyone, except those who have the key to reverse 

17 the watermark. As such, the technique provides the benefit of encryption with 

18 the added benefit of being able to carry extra data in the watermark. 
19 

20 Encryption combined with compression might achieve some of the same effect 

21 as the use of the reversible watermark; however, reversible watermarking can 

22 provide security (you need the watermark key to reverse the watermark and run 

23 the software), extra data capacity (the watermark can carry program related 

24 data), and compressibility (the resulting file after watermarking is 

25 compressible). It is noted that a watermarked file may not be as compressible 

26 as prior to embedding. 
27 

28 There are a variety of ways to increase the size of the payload carried by a 

29 watermark applied in accordance with the present invention. 
30 

31 1 . One can use a triplet of pixels to embed two bits instead of a pair of 

32 pixels to embed one bit. The following reversible transform can be used for 
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1 this purpose: 

2 forward VO = |_1 /3(U0+U 1 +U2)_| 

3 V1 = U2-U1 

4 V2 = U0-U1 
5 

6 reverse U 1 = VO- L1 /3(V1 +V2)J 

7 UO = V2+U1 

8 U2 = V1+U1 
9 

10 2. One can apply the technique to cross spectral components. If R, G and B 

1 1 are the three color component, the following reversible transform can be 

12 used. 
13 

14 forward Y = |_1/4(R+2G+B)J 

15 U = B-G 

16 V=R-G 
17 

1 8 reverse G = Y- L1 /4(U+V)J 

19 R = V+G 

20 B = U+G 



21 

22 3. One can combine (1) and (2) by applying (1) to each color component (row 

23 then column) then apply (2) to the result. 
24 

25 4. One can overlap pairs of pixels or triplets as discussed above to increase the 

26 payload. 
27 

28 The four specific embodiments of the invention described above use a 2x2 

29 pixel region to maximize local other embodiments could use other size regions 

30 such as a 3x3 region etc. 
31 
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1 While the invention has been explained with respect to various embodiments 

2 and alternatives, those skilled in the art will readily realized that a wide array of 

3 alternative embodiments are possible without departing from the spirit, scope 

4 and contribution of this invention. The scope of applicant's invention is limited 

5 only by the appended claims. 
6 

7 
8 
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1 1 . A method of reversibly embedding auxiliary data in a data set 

2 comprising: 

3 transforming the data set from an original domain into transformed data 

4 values with an invertible transform; 

5 expanding selected data values to embed auxiliary data; 

6 inverting the transformed data values, including the data values selected 

7 for expansion, to return the transformed data values to the original domain. 
8 

9 2. The method of claim 1 including: 

10 identifying data values that can be expanded to embed auxiliary data 

1 1 values without causing an underflow or overflow. 
12 

13 3. The method of claim 1 wherein the transformation includes 

14 transforming the data set into fixed and variable values, the variable values 

15 forming a set from which certain transformed data values are selected for 

16 expansion. 
17 

18 4. The method of claim 3 wherein the fixed values remain unchanged 

19 during the auxiliary date embedding operation. 
20 

21 5. The method of claim 3 wherein the fixed values are averages of 

22 selected groups of elements in the data set, and the variable values are 

23 difference values of elements in the selected groups. 
24 

25 6. The method of claim 1 wherein the invertible transform comprises an 

26 integer to integer invertible transform. 
27 

28 7. The method of claim 1 wherein expanding comprises multiplying a 

29 first selected data value by a desired number of states and adding a number 

30 corresponding to a selected state of an auxiliary data value to be embedded in 

31 the first selected data value, and repeating the multiplying and adding for other 

32 data values selected for expansion to embed additional auxiliary data values. 
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1 

2 8. The method of claim 7 including: 

3 identifying data values that can be expanded to embed auxiliary data 

4 values without causing an underflow or overflow. 
5 

6 9. The method of claim 7 wherein the number of states is Iwo, and the 

7 multiplying is performed by shifting bit positions in data values selected for 

8 expansion. 
9 

10 10. The method of claim 1 wherein data values selected for embedding 

1 1 expansion correspond to embedding locations that have a property that is 

12 invariant to changes due to embedding of the auxiliary data, and wherein the 

13 invariant property enables a decoder to identify embedding locations. 
14 

15 11. The method of claim 10 wherein the invariant property is identified 

16 based on whether a data value at an embedding location can be changed to 

17 embed data without causing an underflow or overflow condition. 
18 

19 12. A storage medium on which is stored instructions for performing the 

20 method of claim 1 . 
21 

22 13. The method of claim 1 wherein the invertible transform comprises a 

23 transform to average and difference values, the difference values forming a set 

24 from which values are selected for auxiliary data embedding by expansion. 
25 

26 14. The method of claim 1 wherein the data set comprises an image 

27 signal. 
28 

29 15. The method of claim 1 wherein the transforming, expanding and 

30 inverting is performed repeatedly to data elements at embedding locations 

31 within the data set to embed two or more layers of auxiliary data. 
32 
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1 16. The method of claim 15 wherein each layer has a different decoding 

2 key used to decode the layer. 
3 

4 17. The method of claim 1 wherein expanding includes inserting one or 

5 more extra bits into a selected data value to increase the number of bits after a 

6 most significant, non-zero bit, wherein the auxiliary data is carried in the one or 

7 more extra bits. 
8 

9 18. A method of reading auxiliary data reversibly embedded in a data 

10 set and restoring the data set to the same values as before the reversible 

11 embedding, the method comprising: 

12 transforming the data set from an original domain into transformed data 

13 values with an invertible transform; 

14 extracting auxiliary data from data values previously selected for 

15 embedding of auxiliary data by expansion, including restoring the selected data 

16 values to the same values as before the embedding of the auxiliary data; and 

17 inverting the transformed data values, including the data values selected 

18 for expansion, to return the transformed data values to the original domain. 
19 

20 19. A storage medium on which is stored instructions for performing the 

21 method of claim 18. 
22 

23 20. The method of claim 1 8 wherein one or more bits of the data values 

24 carry auxiliary data, and extracting includes reading the one or more bits of the 

25 data values. 
26 

27 21 . The method of claim 18 including: 

28 identifying data values that have an invariant property to embedding of 

29 auxiliary data to determine which data values are carrying auxiliary embedded 

30 data. 
31 
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1 22. A method of reversibly embedding auxiliary data in a data set 

2 comprising: 

3 selecting embedding locations in the data set that have a property that is 

4 invariant to changes due to embedding of the auxiliary data, and wherein the 

5 invariant property enables a decoder to identify embedding locations; and 

6 reversibly embedding auxiliary data into data values at the embedding 

7 locations. 
8 

9 

10 23. The method of claim 22 including; 

1 1 expanding selected data values to embed auxiliary data. 
12 

13 24. The method of claim 23 wherein the expanding includes inserting 

14 one or more extra bits into a data value to increase the number of bits after a 

15 most significant, non-zero bit, wherein the auxiliary data is carried in the one or 

16 more extra bits. 
17 

18 25. The method of claim 23 wherein expanding includes multiplying a 

19 data value by a number of states and adding a state corresponding to an 

20 auxiliary data value to be embedded. 
21 

22 26. The method of claim 22 wherein the invariant property is identified 

23 based on whether a data value at an embedding location can be changed 

24 without causing an underflow or overflow. 
25 

26 27. A storage medium on which is stored instructions for performing the 

27 method of claim 22. 
28 

29 28. A method of decoding reversibly embedded auxiliary data in a data 

30 set comprising: 

31 identifying a subset of locations in the data set that have a property that 

32 is invariant to changes due to embedding of the auxiliary data; 
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1 extracting auxiliary data from data values at the identified locations; and 

2 restoring values of the data set to the same values as before the 

3 embedding of the auxiliary data into the data set. 
4 

5 29. A storage medium on which is stored instructions for performing the 

6 method of claim 28. 
7 

8 30. The method of claim 28 wherein the auxiliary data is embedded by 

9 expansion of data values. 
10 

11 31 . The method of claim 28 wherein the auxiliary data includes a 

12 location map indicating which of the subset of locations has been embedded 

13 with auxiliary data by expansion. 
14 

15 32. A method of embedding auxiliary data in a data set comprising: 

16 identifying values derived from the data set that are expandable; and 

17 expanding the identified values by inserting an auxiliary data state 

18 corresponding to auxiliary data to be embedded in the identified values. 
19 

20 33. The method of claim 32 wherein the expanding is invertible by 

21 limiting embedding to values that can be expanded without causing an 

22 underflow or overflow. 
23 

24 34. The method of claim 32 wherein the identified values are derived by 

25 exploiting correlation within the data set to compute values that are a function 

26 of the values in the original data set and that are more expandable than the 

27 values in the original data set. 
28 

29 35. The method of claim 32 wherein identified values are chosen for 

30 expansion based on a property that enables the decoder to identify locations of 

31 embedded auxiliary data without using data separate from the data set. 
32 
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1 36. A storage medium on which is stored instructions for performing the 

2 method of claim 32. 
3 

4 37. A method of decoding auxiliary data from an embedded data set 

5 comprising: 

6 identifying values derived from the embedded data set that have been 

7 embedded with auxiliary data; and 

8 extracting auxiliary data from selected values in the embedded data set 

9 that have been embedded with auxiliary data, including extracting inserted 
1 0 auxiliary data state values from the selected values. 

11 

12 38. A storage medium on which is stored instructions for performing the 

1 3 method of claim 37. 
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